Similarity scoring algorithm combining subject synonyms and word vectors

Abstract

The present invention relates to a similarity scoring algorithm combining subject synonyms and word vectors, which is used to realize automatic scoring of geography subjective questions. First, taking geography as the background, a geography dictionary is established by extracting subject knowledge information, and it is introduced into the Word2vec model to train word vectors and construct a geography corpus; then, in view of the problem that the synonym dictionary cannot accurately identify subject synonyms, a geography synonym word library is established; finally, a keyword extraction and weight distribution algorithm is proposed based on part of speech, and the subject knowledge background is integrated into the text similarity calculation, and the credible value of sentence similarity is established based on word similarity, so as to realize the similarity scoring algorithm. The experimental results show that the method is basically consistent with the teacher's scoring trend, and the scoring accuracy rate reaches 88.82%.

Description

Technical Field

[0001] The present invention relates to the fields of natural language processing and machine learning. Background Art

[0002] The automatic scoring technology for subjective questions is the key technology for scoring subjective questions in practice. Existing scoring methods require a large amount of expert-annotated data, including the use of features such as semantic similarity and word weights to build a random forest classifier based on traditional machine learning methods to predict the scores of candidates; the use of a short answer scoring algorithm based on a deep autoencoder to build a scoring model without clearly defining the target answer; and the use of a neural network composed of CNN and LSTM for automatic scoring. In order to realize the automation of online examinations, some researchers have proposed an automatic scoring model for subjective questions based on multi-feature sentence similarity. This matching-based automatic scoring method for subjective questions requires manual design and calculation of complex features such as word form, semantics, and syntax. The matching accuracy is low, and no subject knowledge information is incorporated. The scoring accuracy is quite different from the automatic scoring results of machine learning and deep learning, and the effect is not ideal. Summary of the invention

[0003] In view of the above problems, the present invention ensures the accuracy of subject synonym calculation by constructing a geographical synonym vocabulary, and integrates geographical knowledge information into the corpus, so that the representation of words in the vector space is more in line with the geographical subject background, and combines subject synonyms with word vectors to implement a similarity scoring algorithm. Experiments are carried out using real candidate data from Beijing and Shaanxi Province, and ideal results are achieved, verifying the effectiveness of the algorithm in this paper.

[0004] The similarity scoring algorithm combining subject synonyms and word vectors includes the following steps:

[0005] Step 1: The present invention collects all the texts and some middle school entrance examination questions in the "High School Geography Knowledge Checklist" and "Five-Year College Entrance Examination Three-Year Simulation College Entrance Examination Geography" as geographical knowledge corpus, and performs word segmentation and part-of-speech marking through the Language Technology Platform (LTP), totaling 18,140 words. The sample data is as follows: Figure 1 As shown, some of the words are wrong, such as "due west" and "due south" in adjectives should be marked as directional nouns; "surface" and "meridian" in nouns should be marked as geographical proper nouns. In response to the above problems, this paper analyzes the categories and meanings of modern Chinese parts of speech, and combines the LTP part-of-speech tagging set to manually correct the words with incorrect segmentation or part-of-speech tagging in the processing results of geographical knowledge corpus. After completion, the words and corresponding parts of speech are separated by spaces and written into the geographical dictionary, totaling 13,955 words;

[0006] Step 2: The present invention uses Chinese Wikipedia as the initial corpus, trains word vectors based on the CBOW (Continuous Bag-of-Word) model of Word2vec, and constructs a geographic corpus;

[0007] Step three, using the word vector model trained with Word2vec, can only provide the correlation between words, and the accuracy of semantic similarity calculation is not high. To address this issue, the "Synonym Dictionary" is usually used to improve the accuracy of word semantic similarity calculation, but it is not suitable for identifying synonyms in disciplines. For example, in the "Synonym Dictionary", synonyms for "fragile" include "weak", "cowardly", "spoiled", etc. These are synonyms in people's daily expressions, but in the subject of geography, there is usually an expression such as "ecological fragility", which is replaced by "ecological weakness" and "ecological pampering". The semantics are unclear, the expression is unclear, and the scoring is wrong. To address the problem of inaccurate recognition of subject synonyms, the present invention constructs a geographical synonym dictionary, and the specific steps are as follows;

[0008] a. Read all the words in the geographic dictionary and write them into the geographic synonyms database, with each word occupying one line;

[0009] b. Look up all the synonyms of each word in the geographic dictionary in the Synonym Dictionary and write them after the word in the geographic synonym dictionary, separated by spaces;

[0010] c. Query all similar words of each word in the geographic dictionary in the geographic corpus, and write the words with a similarity above 0.6 after the word in the geographic synonyms lexicon, separated by spaces;

[0011] d. The first word in each line of the geographic synonyms database is taken as the target word, and the following words are taken as candidate words. Based on knowledge bases such as Baidu Encyclopedia, the candidate words in each line are manually screened and supplemented.

[0012] At present, the Geography Thesaurus has compiled a total of 1,843 groups of subject synonyms. Since it was manually constructed without the complete knowledge background of geography experts and there is a certain degree of subjectivity, some subject synonyms may not be precise enough. However, compared with the "Synonym Dictionary", the Geography Thesaurus can identify subject synonyms more accurately.

[0013] Step 4: Extraction and weight allocation of keywords based on part of speech. The present invention proposes a method for extracting and weighting keywords based on part of speech by analyzing the characteristics of college entrance examination geography papers and some middle school test questions and answers, that is, to classify keywords according to part of speech and assign weights to each type of keyword;

[0014] Step 5: Calculate the word similarity based on subject synonyms. In order to realize the semantic similarity calculation of key words in the scoring process, the present invention introduces the concept of subject synonyms, and comprehensively calculates the word similarity based on the subject synonyms provided in the geographical synonym thesaurus, the synonyms provided in the "Tongyi Cilin" and the similar words provided in the geographical corpus;

[0015] Step six, sentence similarity calculation based on word vectors. A sentence is composed of a word or a group of syntactically related words, which contain complex features such as word form, word order, sentence length, and semantics. Any change in the sentence may change the sentence semantics, making it difficult for the computer to understand the sentence. This paper draws on the sentence similarity calculation method proposed in the literature, uses the word vectors provided in the geographic corpus, and weights the word vectors to construct sentence vectors through the keyword extraction and weight allocation algorithm in step four. The cosine similarity algorithm is used to calculate the similarity of the two sentence vectors to obtain the sentence similarity;

[0016] Step 7: Combine the subject synonyms and word vector similarity scoring algorithm. This paper ensures the accuracy of word similarity calculation by constructing a geographic synonym database, but the accuracy of sentence similarity calculation is not high at present, especially two unrelated sentences will also calculate a certain similarity value, resulting in inaccurate scoring. To address this problem, word similarity calculation is used as the standard to establish a credible value for sentence similarity. Based on the word similarity and sentence similarity between the candidate's answer and the standard answer, the comprehensive similarity can be calculated and automatically scored. In order to meet the teacher's review method, a scoring coefficient is formulated based on the comprehensive similarity. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Building processes for geographic corpora;

[0018] Figure 2 This is a comparison of the scoring effects of Experiment 1 of the present invention;

[0019] Figure 3 The scoring effect of Experiment 2 of the present invention Figure 1 ;

[0020] Figure 4 The scoring effect of Experiment 2 of the present invention Figure 2 ;

[0021] Figure 5 The scoring accuracy of Experiment 2 of the present invention is Figure 1 ;

[0022] Figure 6 The scoring accuracy of Experiment 2 of the present invention is Figure 2 . DETAILED DESCRIPTION

[0023] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0024] The process of the method involved in the present invention includes the following steps:

[0025] (1) Geographic dictionary

[0026] The present invention collected all the texts in "High School Geography Knowledge List" and "Five-Year College Entrance Examination and Three-Year Simulation · Geography for College Entrance Examination", as well as some middle school examination questions, as geographical knowledge corpus. After word segmentation and part-of-speech tagging through the Language Technology Platform (LTP), there are a total of 18,140 words. The sample data is as Figure 1 shown. Some of the words are incorrect. For example, "due west" and "due south" in adjectives should be tagged as direction nouns; words such as "earth's surface" and "meridian" in nouns should be tagged as geographical proper nouns. In response to the above problems, this paper analyzed the categories and meanings of modern Chinese parts of speech, and combined with the LTP part-of-speech tagging set, manually corrected the words with incorrect word segmentation or part-of-speech tagging in the processing results of the geographical knowledge corpus. After completion, the words and their corresponding parts of speech are separated by spaces and written into the geographical dictionary, with a total of 13,955 words.

[0027] (2) Geographic corpus

[0028] This paper uses Chinese Wikipedia as the initial corpus, trains word vectors based on the CBOW (Continuous Bag-of-Word) model of Word2vec, and constructs a geographic corpus.

[0029] Since LTP incorporates a dictionary strategy, introducing the geographic dictionary during word segmentation can more accurately identify words in the geography discipline. Comparing the word vector relationships before and after introducing the geographic dictionary, among them, "cold front" belongs to geographical discipline words, and "promote" belongs to natural language, not geographical discipline words. It can be seen that introducing the geographic dictionary during the process of training word vectors changes the context of the words in each text of Chinese Wikipedia, resulting in a change in the position of the words in the high-dimensional vector space, which has a greater impact on the similarity relationship between geographical discipline words, and adds a large number of geographical discipline-related words, making the constructed corpus more tend to be a corpus in the geographical discipline field. Therefore, it is used as the basic resource for carrying geographical knowledge.

[0030] (3) Geographic synonym thesaurus

[0031] The word vector model trained by Word2vec can only provide the correlation between words, and the accuracy of semantic similarity calculation is not high. To address this problem, the "Synonym Dictionary" is usually used to improve the accuracy of word semantic similarity calculation, but it is not suitable for identifying synonyms in disciplines. For example, in the "Synonym Dictionary", the synonyms of "fragile" include "weak", "cowardly", "spoiled", etc. These are synonyms in people's daily expressions, but in the subject of geography, there is usually an expression such as "ecological fragility", which is replaced by "ecological weakness" and "ecological pampered". The semantics are unclear and the expression is unclear, resulting in scoring errors. To address the problem of inaccurate recognition of subject synonyms, this paper constructs a geographical synonym dictionary. The specific steps are as follows;

[0032] a. Read all the words in the geographic dictionary and write them into the geographic synonyms database, with each word occupying one line;

[0033] b. Look up all the synonyms of each word in the geographic dictionary in the Synonym Dictionary and write them after the word in the geographic synonym dictionary, separated by spaces;

[0034] c. Query all similar words of each word in the geographic dictionary in the geographic corpus, and write the words with a similarity above 0.6 after the word in the geographic synonyms lexicon, separated by spaces;

[0035] d. The first word in each line of the geographic synonyms database is taken as the target word, and the following words are taken as candidate words. Based on knowledge bases such as Baidu Encyclopedia, the candidate words in each line are manually screened and supplemented.

[0036] At present, the Geography Thesaurus has compiled a total of 1,843 groups of subject synonyms. Since it was manually constructed without the complete knowledge background of geography experts and there is a certain degree of subjectivity, some subject synonyms may not be precise enough. However, compared with the "Synonym Dictionary", the Geography Thesaurus can identify subject synonyms more accurately.

[0037] (4) Keyword extraction and weight assignment based on part of speech

[0038] Keyword extraction and weight allocation based on part of speech. The present invention proposes a keyword extraction and weight allocation method based on part of speech by analyzing the characteristics of college entrance examination geography papers and some middle school test questions and answers, that is, the keywords are classified according to part of speech, and each type of keyword is assigned the following weight:

[0039]

[0040] The answer text is segmented and marked with part of speech through LTP and geographic dictionary. Suppose the number of A-type words in the text is a, and the weight of each word is w a , the number of words in category B is b, and the weight of each word is wb , the number of words in category C is c, and the weight of each word is w c , taking the weight of category B keywords as the benchmark x, the keyword weight calculation equation is shown in Formula 1:

[0041]

[0042] By deducing equation 1, the weight calculation of category A keywords is shown in equation 2:

[0043]

[0044] The weight calculation of category B keywords is shown in formula 3:

[0045]

[0046] The weight calculation of category C keywords is shown in formula 4:

[0047]

[0048] The keyword extraction and weight assignment algorithm is as follows:

[0049] a. Algorithm input: target text S;

[0050] b. Based on the geographic dictionary, use LTP to segment the text S and mark the parts of speech. Refer to the keyword weight distribution and extract keywords according to the part of speech to obtain the sequence Seq(x,t)={(x1,t1),(x2,t2),(x3,t3),...,(x n ,t n )}, where x represents a word, t represents the part of speech marked for word x, and n represents the number of words.

[0051] c. Traverse Seq(x, t) and count the number of words in category A, the number of words in category B, and the number of words in category C according to the keyword weight distribution;

[0052] d. Calculate the A-type part-of-speech weight w by formula 2 a , calculate the B-type part-of-speech weight w by formula 3 b , calculate the C-type part-of-speech weight w by formula 4 c .

[0053] e. Traverse Seq(x,t), refer to the keyword weight distribution to determine the category to which the part of speech t belongs, assign the corresponding weight w to the word x according to the category, and obtain the sequence Seq(x,t,w)={(x1,t1,w1),(x2,t2,w2),(x3,t3,w3),...,(x n ,t n ,w n )}.

[0054] f. Output: Seq(x,t,w)

[0055] (5) Word similarity calculation based on subject synonyms

[0056] In order to realize the semantic similarity calculation of key words in the scoring process, the present invention introduces the concept of subject synonyms, and comprehensively calculates the word similarity based on the subject synonyms provided in the geographical synonym thesaurus, the synonyms provided in the "Synonym Cilin" and the similar words provided in the geographical corpus. The specific principle is as follows:

[0057] 1) Algorithm input: words Word1 and Word2;

[0058] 2) Initialization: geographic synonym thesaurus dlSym, list sym coded as "=" in the Synonym Dictionary, geographic corpus dlmodel;

[0059] 3) Traverse dlSym and query the list dlSymList of Word1. If the length of dlSymList is greater than 0, go to 4); otherwise, go to 5);

[0060] 4) Traverse dlSymList and check whether Word2 exists. If so, wordSim(Word1, Word2)=1, go to 9), otherwise, go to 5);

[0061] 5) Traverse sym and query the list symList of Word1. If the length of symList is greater than 0, go to 6); otherwise, go to 7;

[0062] 6) Traverse symList and check whether Word2 exists. If so, wordSim(Word1, Word2) = 0.8, go to 9), otherwise, go to 7);

[0063] 7) Check whether Word1 exists in dlmodel. If so, go to 8). Otherwise, wordSim(Word1, Word2)=0, go to 9);

[0064] 8) Check whether Word2 exists in dlmodel. If so, calculate the similarity between Word1 and Word2 through dlmodelSym(Word1, Word2), then wordSim(Word1, Word2) = dlmodelSym(Word1, Word2) × 0.6, to 9), otherwise wordSim(Word1, Word2) = 0, to 9);

[0065] 9) Output word similarity: wordSim(Word1,Word2).

[0066] (6) Sentence similarity calculation based on word vectors

[0067] The present invention uses a sentence similarity calculation method, uses the word vectors provided in the geographic corpus, and weights the word vectors to construct sentence vectors through keyword extraction and weight distribution algorithms. The cosine similarity algorithm is used to calculate the similarity of two sentence vectors to obtain sentence similarity. The specific algorithm is as follows:

[0068] a. Algorithm input: statements S1 and S2;

[0069] b. Initialization: geographic corpus dlmodel;

[0070] c. Extract keywords and assign weights to S1 and S2 using the method in step (4) to obtain sequences Seq1 and Seq2;

[0071] d. Traverse each word in Seq1 and Seq2, and check whether the word exists in dlmodel. If so, use dlmodel to convert the word into a word vector (the word vector dimension is 192). Otherwise, record the word as a 0 vector.

[0072] e. The word vector of the i-th word in the sequence Seq is wv i , weight is w i , Seq1 and Seq2 are calculated by formula 5 to obtain sentence vectors V1 and V2; n represents the number of words.

[0073]

[0074] f. Using the cosine similarity algorithm, calculate the cosine similarity cosSim(V1,V2) of V1 and V2 through formula 6, so that the sentence similarity sentenceSim(S1,S2) of S1 and S2 is equal to cosSim(V1,V2);

[0075]

[0076] g. Output sentence similarity: sentenceSim(S1,S2).

[0077] (7) Similarity scoring algorithm combining subject synonyms and word vectors

[0078] The present invention ensures the accuracy of word similarity calculation by constructing a geographic synonym vocabulary, but the accuracy of sentence similarity calculation is not high at present, especially two unrelated sentences will also calculate a certain similarity value, resulting in inaccurate scoring. To address this problem, based on word similarity calculation as the standard, a credible value α is established for sentence similarity, as shown below.

[0079]

[0080]

[0081] Based on the word and sentence similarity between the examinee's answer and the standard answer, the comprehensive similarity can be calculated for automatic scoring. In order to meet the teacher's review method, a scoring coefficient λ is formulated based on the comprehensive similarity. This coefficient can be manually formulated by the question setting experts to formulate a more accurate scoring method for questions of different difficulty coefficients. The corresponding relationship between similarity and scoring coefficient λ is as follows:

[0082]

[0083] Combining the similarity scoring algorithm of subject synonyms and word vectors, the specific steps are as follows:

[0084] (1) Algorithm input: standard answer text B, corresponding score score, and candidate answer text S;

[0085] (2) Extract keywords and assign weights to B and S using the method in step 1 to obtain the sequence

[0086] SeqB(bx,bt,bw)={(bx1,bt1,bw1),(bx2,bt2,bw2),...,(bx n ,bt n ,bw n )},

[0087] SeqS(sx,st,sw)={(sx1,st1,sw1),(sx2,st2,sw2),...,(sx n ,st n ,sw n )};

[0088] (3) Traverse SeqB(bx,bt,bw) and find the part of speech bt of the i-th word in SeqS(sx,st,sw) that matches SeqB(bx,bt,bw) i The same word set SWList(sx) = {sx1, sx2, ..., sx n};

[0089] (4) Use the algorithm in step 2 to convert bx i Calculate the similarity with the words in SWList(sx) in turn, and multiply the maximum value of the calculation result by the weight bw i Get the weighted similarity s of the words i , then the word similarity between B and S is wordSim(B,S)=∑s i ;

[0090] (5) Calculate B and S using the sentence similarity algorithm in step 3 to obtain the sentence similarity between B and S, sentenceSim(B,S);

[0091] The corresponding relationship between the sentence similarity credibility value α and the word similarity is as follows:

[0092]

[0093] (6) Referring to the above correspondence, the sentence similarity credibility value α is calculated by wordSim(B,S), and the comprehensive similarity similarity(B,S) between B and S is calculated by using formula 3;

[0094] similarity(B,S)=wordSim(B,S)+sentenceSim(B,S)×α (3)

[0095] (7) Referring to the correspondence between similarity and scoring coefficient, the corresponding scoring coefficient λ is calculated by similarity(B, S), and the student score studentScore is calculated using Formula 4;

[0096] studentScore=score×λ (4)

[0097] (8) Output the student’s score: studentScore.

Claims

1. A similarity scoring algorithm combining subject synonyms and word vectors, characterized in that: The steps include: Step 1: keyword extraction and weight assignment based on part of speech; Step 2: Calculate the word similarity based on subject synonyms; Step 3: Calculate sentence similarity based on word vectors; Step 4: Combine subject synonyms with word vector similarity scoring algorithm; The keyword extraction and weight allocation based on part of speech described in step 1 are as follows: The keyword extraction algorithm is as follows: a. Algorithm input: target text S; b. Based on the geographic dictionary, use LTP to segment the text S and mark the parts of speech. Refer to the weight distribution and extract keywords according to the parts of speech to obtain the sequence Seq(x, t) = {(x1, t1), (x2, t2), (x3, t3), ..., (x n , t n )}, where x represents a word, t represents the part of speech marked on word x, and n represents the number of words. c. Traverse Seq(x, t) and count the number of words in category A, the number of words in category B, and the number of words in category C according to the weight distribution; d. Calculate the weight w of type A a , calculate the part-of-speech weight w of class B b , calculate the C-type part-of-speech weight w c ; e. Traverse Seq(x, t), refer to the weight distribution to determine the category to which the part of speech t belongs, assign the corresponding weight w to the word x according to the category, and obtain the sequence Seq(x,t,w)={(x1,t1,w1),(x2,t2,w2),(x3,t3,w3),...,(x n ,t n ,w n )}; f. Output: Seq(x, t, w); Step 2 is as follows: 1) Algorithm input: words Word1 and Word2; 2) Initialization: geographic synonym thesaurus dlSym, list sym coded as "=" in the Synonym Dictionary, geographic corpus dlmodel; 3) Traverse dlSym and query the list dlSymList of Word1. If the length of dlSymList is greater than 0, go to 4); otherwise, go to 5); 4) Traverse dlSymList and check whether Word2 exists. If so, wordSim(Word1, Worrd2)=1, go to 9), otherwise, go to 5); 5) Traverse sym and query the list symList of Word1. If the length of symList is greater than 0, go to 6); otherwise, go to 7; 6) Traverse symList and check whether Word2 exists. If so, wordSim(Word1, Word2)=0.8, go to 9), otherwise, go to 7); 7) Check whether Word1 exists in dlmodel. If so, go to 8). Otherwise, wordSim(Word1, Word2)=0, go to 9); 8) Check whether Word2 exists in dlmodel. If so, calculate the similarity between Word1 and Word2 through dlmodelSym(Word1, Word2), then wordSim(Word1, Word2)=dlmodelSym(Word1, Word2)×0.6, to 9), otherwise wordSim(Word1, Word2)=0, to 9); 9) Output word similarity: wordSim(Word1, Word2); The sentence similarity based on word vectors described in step 3 is calculated as follows: a. Algorithm input: statements S1 and S2; b. Initialization: geographic corpus dlmodel; c. Extract keywords and assign weights to S1 and S2 using the method in step 1 to obtain sequences Seq1 and Seq2; d. Traverse each word in Seq1 and Seq2, and check whether the word exists in dlmodel. If so, use dlmodel to convert the word into a word vector (the word vector dimension is 192). Otherwise, record the word as a 0 vector. e. The word vector of the i-th word in the sequence Seq is wv i , weight is w i , Seq1 and Seq2 are calculated by formula 1 to obtain sentence vectors V1 and V2; f. Using the cosine similarity algorithm, calculate the cosine similarity cosSim(V1, V2) of V1 and V2 through formula 2, so that the sentence similarity sentenceSim(S1, S2) of S1 and S2 is equal to cosSim(V1, V2); g. Output sentence similarity: sentenceSim(S1, S2); Step 4 is as follows: (1) Algorithm input: standard answer text B, corresponding score score, and candidate answer text S; (2) Extract keywords and assign weights to B and S using the method in step 1, and obtain the sequence SeqB(bx, bt, bw) = {(bx1, bt1, bw1), (bx2, bt2, bw2), ..., (bx n , bt n , bw n )}, SeqS(sx, st, sw={(sx1, st1, sw1), (sx2, st2, sw2),..., (sx n , st n , sw n )}; (3) Traverse SeqB(bx, bt, bw) and find the part of speech bt of the i-th word in SeqS(sx, st, sw) that matches SeqB(bx, bt, bw) i The same word set SWList(sx) = {sx1, sx2, ..., sx n }; (4) Use the algorithm in step 2 to convert bx i Calculate the similarity with the words in SWList(sx) in turn, and multiply the maximum value of the calculation result by the weight bw i Get the weighted similarity s of the words i , then the word similarity between B and S is wordSim(B, S) = ∑s i ; (5) Calculate B and S using the sentence similarity algorithm in step 3 to obtain the sentence similarity between B and S, sentenceSim(B, S); The corresponding relationship between the sentence similarity credibility value α and the word similarity is as follows: (6) Referring to the above correspondence, the sentence similarity credibility value α is calculated by wordSim(B, S), and the comprehensive similarity similarity(B, S) between B and S is calculated by using formula 3; similaity(B,S)=wordSim(B,S)+sentenceSim(B,S)×α (3) The corresponding relationship between similarity and scoring coefficient λ is as follows: (7) Referring to the correspondence between similarity and scoring coefficient, the corresponding scoring coefficient λ is calculated by similarity(B, S), and the student score studentScore is calculated using Formula 4; studentScore=score×λ (4) (8) Output the student’s score: studentScore.

Images