A scoring method for keyword hits based on word vectors

Through the scoring method of keyword hitting based on word vectors, the problem of inconsistent dimensional characteristics of open-question-type content is solved, more accurate scoring and guidance feedback are achieved, and content scoring is supported in multiple scenarios.

CN114372472BActive Publication Date: 2024-11-12SUZHOU CHIVOX INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111440270.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-11-30
Publication Date
2024-11-12
Estimated Expiration
2041-11-30

AI Technical Summary

Technical Problem

The content dimension characteristics of the existing open-question type are inconsistent in terms of keyword matching hit rate and semantic similarity, resulting in low hit rate but high similarity, and requires a large number of labeling and scoring, which is not conducive to model optimization.

Method used

The keyword hit score method based on word vectors is used to match the keywords in the sample with the fragments of the freely recognized text, calculate the degree of match, obtain the score of each keyword key point, and perform a weighted average to obtain the overall semantic match. Score, using formulas to represent the degree of match and weighted average process.

Benefits of technology

It achieves more direct and accurate rating feedback, supports content ratings in multiple scenarios, and can more accurately help students evaluate their expression integrity in content, and provide guidance feedback other than rating.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114372472B_ABST
    Figure CN114372472B_ABST
Patent Text Reader

Abstract

The invention discloses a scoring method for keyword hits based on word vectors, comprising the following steps: S1: matching keywords in a sample with segments of freely identifiable text to obtain a matching degree; S2: obtaining a score for each keyword key point through the matching degree; S3: weighted averaging the score for each keyword key point to obtain a group of overall semantic matching scores for the sample for keyword key points. The invention relates to content evaluation feedback and scoring for relatively open question types such as open questions and essay questions in oral examinations. The content scoring method can support content scoring for a variety of scenarios, namely, content scoring for only reference answers, content scoring for only keywords, and content scoring for input keywords + reference answers. Compared with previous content scoring algorithms, the invention can provide more direct and accurate scoring feedback, thereby not only being able to more accurately help students evaluate the completeness of their own expression in terms of content, but also providing guidance feedback in addition to scoring.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of speech signal processing, and in particular to a keyword hit scoring method based on word vectors. Background Art

[0002] As an important medium of interpersonal communication, spoken language occupies an extremely important position in real life. With the continuous development of social economy and the trend of economic globalization, people have put forward higher and higher requirements on the efficiency of language learning and the objectivity, fairness and large-scale testing of language assessment. Open questions in oral examinations, such as oral composition, story retelling and picture description, are important questions that reflect the oral expression ability of candidates. The content evaluation and scoring of open questions need to be evaluated according to the content dimension characteristics.

[0003] The content dimension features of existing open questions are keyword matching hit rate, semantic similarity based on Word Move Distence, text vector similarity features based on inverse document frequency weight-word vector, etc., and then the scoring model is trained with the labeled data for scoring. However, according to the above scoring method, there will be the following problems: there is inconsistency between semantic similarity and keyword hit rate, that is, the hit rate may not be high but the similarity is very high; a large number of labeled scores are required, which is not conducive to model optimization. Therefore, there is still a need to improve the content dimension scoring of the multi-dimensional algorithm for open questions. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiencies of the prior art and to provide a scoring method for keyword hits based on word vectors that can give more direct and accurate scoring feedback and have more precise scoring.

[0005] To achieve the above object, the technical solution adopted by the present invention is: a scoring method for keyword hits based on word vectors, comprising the following steps:

[0006] S1: Match the keywords in the sample with the fragments of the freely identifiable text to obtain the matching degree;

[0007] S2: Get the score of each keyword point by the matching degree;

[0008] S3: Take a weighted average of the scores of each keyword key point to obtain the overall semantic matching score of a set of keyword key points for the sample.

[0009] Furthermore, the score of the matching degree is expressed by the following formula:

[0010] Among them, KEY k,yThe yth keyword representing the kth keyword key point, k = 1, 2, ..., y = 1, 2, ...;

[0011] WindowRec m,i represents the text segment in the i-th window of the recognized text of the m-th document, m=1, 2, ..., i=1, 2, ...;

[0012] WordMatch(KEY k,y ,WindowRec m,i ) represents the character matching score between the keyword and the free recognition text;

[0013] R stem *StemWordMatch(KEY k,y ,WindowRec m,i ) indicates the keyword KEY k,y With WindowRec m,i The stem matching score, R stem is the overall adjustment factor for the score;

[0014] ∑ j=1 a k,j *f(KEY k,y,j ,WindowRec m,i ) represents the semantic matching score between the keyword and the free recognition text;

[0015] F(KEY k,y ,WindowRec m,i ) takes the maximum value among the three expressions;

[0016] F(KEY k,y ,WindowRec m,i )∈[0,1], 1 means a perfect match, 0 means a perfect mismatch.

[0017] Furthermore,

[0018] WordMatch(KEY k,y ,WindowRec m,i )=1 / length(KEY k,y )*∑ j [KEY k,y,j ,WindowRec m,i ] Among them, KEY k,y,j Indicates the jth word in the yth keyword or phrase in the kth keyword point; [KEY k,y,j ,WindowRec m,i ] indicates whether a word in the keyword key point appears in the fragment, if yes, it is 1, otherwise it is 0.

[0019] Furthermore, StemWordMatch(KEY k,y ,WindowRec m,i )=1 / length(KEY k,y )*∑ j=1 [stem(KEY k,y,j ),stem(WindowRec m,i )] Among them, stem(KEY k,y,j ) means to get KEY k,y,j The word stem is calculated using the PorterStemmer algorithm.

[0020] Furthermore, a k,j represents the weight value of the jth word in the kth keyword point, j = 1, 2, ...;

[0021]

[0022] Through the cosine function cos(KEY k,y,j ,WindowRec m,i ) is compared with the judgment threshold thr to obtain the matching score between the word and the recognized text segment. If the cosine function cos(KEY k,y,j ,WindowRec m,i ) is greater than the judgment threshold thr, it is considered a match, and the corresponding word-level semantic match score is 1, otherwise it is 0;

[0023] Furthermore, the following formula is used in S2 to obtain the score of each keyword point:

[0024] S k =Max i Max y [F(KEY k,y ,WindowRec m,i )]; where S k ∈[0, 1] represents the score of the sample on the kth keyword point; if the content is fully expressed, the score is 1, and if it is completely irrelevant, the score is 0;

[0025] Max i Indicates the maximum value of the keyword score on all segments of the sample, Max y Indicates the maximum matching score of all similar words or phrases of the keyword key point.

[0026] Furthermore, S3 uses the following formula to perform a weighted average of the key points scores of each keyword:

[0027]

[0028] Among them, KeyScore m ∈[0, 1] represents the keyword key point matching score of the mth sample, where a complete match is 1 point and a complete mismatch is 0 point;

[0029] c k represents the weight value of the kth key point; k represents the number of keyword key points contained in the sample, k = 1, 2, ...

[0030] Furthermore, in step S1, the keywords include at least one of self-defined keywords and generated keywords; for the generated keywords, stop words are first defined, and then all words in the reference answer that are not stop words are used as generated keywords.

[0031] Furthermore, in step S3, if it is a custom keyword, c k Take 1. If it is a generated keyword, take the IDF value; where IDF is the inverse document frequency.

[0032] Further, in step S3, the keyword key point matching score KeyScore is obtained. m Afterwards, the final content score is obtained by the following formula:

[0033]

[0034] Among them, FinalScore m ∈[0, 1] represents the content score of the mth sample, with a full score of 1 and a completely irrelevant score of 0;

[0035] KeyGenScore represents the content score calculated based on the keywords generated from the reference answer;

[0036] KeyDefscore indicates the content score calculated based on the keywords passed in by the user;

[0037] FinalScore m Indicates the final score result of the mth student;

[0038] If the keywords in the scoring process only include generated keywords, the final content score is KeyGenScore; if the keywords in the scoring process only include custom keywords, the final content score is KeyDefscore;

[0039] If the keywords in the scoring process include custom keywords and generated keywords;

[0040] When KeyGenScore-KeyDefscore<0.5, the final content score is KeyDefscore;

[0041] When KeyGenScore-KeyDefscore>0.5, the final content score is

[0042] Due to the application of the above technical solution, the present invention has the following advantages compared with the prior art:

[0043] The present invention discloses a scoring method for keyword hits based on word vectors, which involves content evaluation feedback and scoring of relatively open questions such as open questions and essay questions in oral examinations. The content scoring method can support content scoring in multiple scenarios, namely, content scoring of reference answers only, content scoring of keywords only, and content scoring of keywords + reference answers. Compared with previous content scoring algorithms, the present invention can give more direct and accurate scoring feedback, which can not only help students evaluate the completeness of their expression in content more accurately, but also give guiding feedback in addition to scoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The technical solution of the present invention is further described below in conjunction with the accompanying drawings:

[0045] Attached Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0046] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation of the present invention.

[0047] Please see attached Figure 1 The present invention discloses a scoring method for keyword hits based on word vectors, comprising the following steps: S1: matching keywords in a sample with segments of freely identifiable text to obtain a matching degree; S2: obtaining a score for each keyword key point through the matching degree; S3: weighted averaging the score for each keyword key point to obtain an overall semantic matching score for a group of keyword key points for the sample.

[0048] Specifically, the formula for matching the keywords in the sample with the fragments of the freely identifiable text to obtain the matching degree in S1 is as follows:

[0049]

[0050] The fragments of freely identifiable text are obtained by sliding the window with a certain step size and window length. The matching methods include character matching, stem matching and semantic matching. The sample refers to the answer of a certain student.

[0051] F(KEY k,y ,WindowRecm,i ) represents the measurement of the matching score between two strings. The maximum value of the three expressions is the score of a specific keyword.

[0052] F(KEY k,y ,WindowRec m,i )∈[0,1], 1 means a perfect match, 0 means a perfect mismatch.

[0053] Among them, KEY k,y The yth keyword representing the kth keyword key point, k = 1, 2, ..., y = 1, 2, ..., all keywords in the set are synonyms and are equivalent to each other; WindowRec m,i Represents the keywords in the i-th window of the recognition text of the m-th document, m=1, 2, ..., i=1, 2, ...

[0054] In addition, WordMatch(KEY k,y ,WindowRec m,i ) represents the character matching score between the keyword and the free recognition text; R stem *StemWordMatch(KEY k,y ,WindowRec m,i ) indicates the keyword KEY k,y With WindowRec m,i The stem matching score, R stem is the overall adjustment coefficient of the score; ∑ j=1 a k,j *f(KEY k,y,j ,WindowRec m,i ) represents the semantic matching score between the keyword and the free recognition text.

[0055] Specifically:

[0056] WordMatch(KEY k,y ,WindowRec m,i )=1 / length(KEY k,y )*∑ j [KEY k,y,j ,WindowRec m,i ] Among them, KEY k,y,j Indicates the jth word in the yth keyword or phrase in the kth keyword point; [KEY k,y,j ,WindowRec m,i ] indicates whether a word in the keyword key point appears in the fragment, if yes, it is 1, otherwise it is 0.

[0057] StemWordMatch(KEYk,y ,WindowRec m,i )=1 / length(KEY k,y )*∑ j=1 [stem(KEY k,y,j ),stem(WindowRec m,i )] Among them, stem(KEY k,y,j ) means to get KEY k,y,j The stemming process is performed by using the PorterStemme algorithm. For example, the stem of the word happiness is happy.

[0058] In addition, a k,j Represents the weight value of the jth word in the kth keyword point, j=1, 2,….

[0059]

[0060] Through the cosine function cos(KEY k,y,j ,WindowRec m,i ) is compared with the judgment threshold thr to obtain the matching score between the word and the recognized text segment. If the cosine function cos(KEY k,y,j ,WindowRec m,i ) is greater than the judgment threshold thr, it is considered a match and the corresponding word-level semantic match score is 1, otherwise it is 0.

[0061] In step S2, the score of each keyword point is obtained using the following formula:

[0062] S k =Max i Max y [F(KEY k,y ,WindowRec m,i )]; where keywords may contain synonyms, the above formula is also the score after comprehensive consideration of all synonymous keywords, that is, calculating the comprehensive score of a set of keywords that are synonymous with the keyword in the sample.

[0063] Among them, S k ∈[0, 1] represents the score of the sample on the kth keyword point; if the content is fully expressed, the score is 1, and if it is completely irrelevant, the score is 0.

[0064] S k =Max i Max y [F(KEY k,y ,WindowRec m,i)], the first Max from the left represents the maximum value of the keyword score on all fragments of the sample, and the second Max represents the maximum value of the matching scores of all keywords that are synonymous with the keyword.

[0065] At the same time, in step S3, the following formula is used to perform weighted average on the key points score of each keyword:

[0066]

[0067] Among them, KeyScore m ∈[0, 1] represents the keyword key point matching score of the mth sample, where a complete match is 1 point and a complete mismatch is 0 point; c k represents the weight value of the kth key point; k represents the number of keyword key points contained in the sample, k = 1, 2, ...

[0068] As a further preferred embodiment, in step S1, the keywords include at least one of self-defined keywords and generated keywords; for the generated keywords, stop words are first defined, and then all words in the reference answer that are not stop words are used as generated keywords.

[0069] As a further preferred embodiment, in step S3, if it is a custom keyword, c k Take 1. If it is a generated keyword, take the IDF value, where IDF is the inverse document frequency.

[0070] As a further preferred embodiment, in step S3, the keyword key point matching score KeyScore is obtained. m Afterwards, the final content score is obtained by the following formula:

[0071]

[0072] Among them, FinalScore m ∈[0, 1] represents the content score of the mth sample, with a full score of 1 and a completely irrelevant score of 0;

[0073] KeyGenScore represents the content score calculated based on the keywords generated from the reference answer;

[0074] KeyDefscore indicates the content score calculated based on the keywords passed in by the user;

[0075] FinalScore m Indicates the final score result of the mth student;

[0076] If the keywords in the scoring process only include generated keywords, the final content score is KeyGenScore; if the keywords in the scoring process only include custom keywords, the final content score is KeyDefscore;

[0077] If the keywords in the scoring process include custom keywords and generated keywords;

[0078] When KeyGenScore-KeyDefscore<0.5, the final content score is KeyDefscore;

[0079] When KeyGenScore-KeyDefscore>=0.5, the final content score is

[0080] The present invention discloses a scoring method for keyword hits based on word vectors, which involves content evaluation feedback and scoring of relatively open questions such as open questions and essay questions in oral examinations. The content scoring method can support content scoring in multiple scenarios, namely, content scoring of only reference answers, content scoring of only keywords, and content scoring of input keywords + reference answers. Compared with previous content scoring algorithms, the present invention can give more direct and accurate scoring feedback, thereby not only helping students to more accurately evaluate the completeness of their expression in content, but also giving guiding feedback in addition to scoring.

[0081] The above are only specific application examples of the present invention and do not constitute any limitation on the protection scope of the present invention. Any technical solution formed by equivalent transformation or equivalent replacement shall fall within the protection scope of the present invention.

Claims

1. A scoring method for keyword hits based on word vectors, characterized in that: The steps include: S1: Match the keywords in the sample with the fragments of the freely identifiable text to obtain the matching degree; S2: Get the score of each keyword point by the matching degree; S3: Perform weighted average on the scores of each keyword key point to obtain the overall semantic matching score of a group of keyword key points for the sample; The matching score is expressed as follows: ;in, The yth keyword representing the kth keyword key point, , ; represents the text segment within the i-th window of the recognized text of the m-th document, , ; Indicates the character matching score between the keyword and the free recognition text; Indicates keywords and The stem matching score of is the overall adjustment factor for the score; Indicates the semantic matching score between the keyword and the free recognition text; Take the maximum value among the three expressions; , 1 means a complete match, 0 means no match at all; In S3, the following formula is used to perform a weighted average of the key points scores of each keyword: ; in, Indicates the keyword key point matching score of the mth sample, where a complete match is 1 point and a complete mismatch is 0 point; Indicates The weight of each point; Indicates the number of key words contained in the sample. ; In step S3, the keyword key point matching score is obtained. Afterwards, the final content score is obtained by the following formula: in, It represents the content score of the mth sample, with a full score of 1 and a completely irrelevant score of 0; KeyGenScore represents the content score calculated based on the keywords generated from the reference answer; KeyDefscore indicates the content score calculated based on the keywords passed in by the user; Indicates the final score result of the mth student; If the keywords in the scoring process only include generated keywords, the final content score is KeyGenScore; if the keywords in the scoring process only include custom keywords, the final content score is KeyDefscore; If the keywords in the scoring process include custom keywords and generated keywords; When KeyGenScore-KeyDefscore<0.5, the final content score is KeyDefscore; When KeyGenScore-KeyDefscore >=0.5, the final content score is .

2. The keyword hit scoring method based on word vectors according to claim 1, characterized in that: in, represents the jth word in the yth keyword or phrase in the kth keyword point; Indicates whether a word in the keyword key point appears in the segment, if yes, it is 1, otherwise it is 0.

3. The keyword hit scoring method based on word vectors as claimed in claim 1, characterized in that: in, Indicates taking The word stem is calculated using the Porter Stemmer algorithm.

4. The keyword hit scoring method based on word vectors as claimed in claim 1, characterized in that: represents the weight value of the jth word in the kth keyword key point, ; Through the cosine function and judgment threshold By comparison, we get the matching score between the word and the segment of the recognized text. If the cosine function If it is greater than the judgment threshold thr, it is considered a match, and the corresponding word-level semantic match score is 1, otherwise it is 0.

5. The keyword hit scoring method based on word vector as claimed in claim 1, characterized in that: In S2, the following formula is used to obtain the score of each keyword point: ;in, Indicates that the sample is If the content is fully expressed, the score is 1 point; if it is completely irrelevant, the score is 0; It represents the maximum value of the keyword score on all the fragments of the sample. Indicates the maximum matching score of all similar words or phrases of the keyword key point.

6. The keyword hit scoring method based on word vectors as claimed in claim 1, characterized in that: In step S1, the keywords include at least one of self-defined keywords and generated keywords; for the generated keywords, stop words are first defined, and then all words in the reference answer that are not stop words are used as generated keywords.

7. The keyword hit scoring method based on word vectors as claimed in claim 1, characterized in that: In step S3, if it is a custom keyword, Take 1. If it is a generated keyword, take the IDF value; where IDF is the inverse document frequency.