A method for evaluating the quality of scientific research paper abstract generation and related equipment

Abstract

The present invention provides a method for evaluating the quality of scientific research paper abstract generation and related equipment, which obtains the original text, target abstract and references of a target scientific research paper and performs bidirectional encoding in sequence to obtain an original text embedding vector, an abstract text embedding vector and a reference text embedding vector, extracts a pseudo target abstract based on the original text and encodes it to obtain a pseudo abstract text embedding vector; calculates each text embedding vector to obtain a comprehensive semantic evaluation score; and calculates a natural coherence score based on the calculated word-by-word probability and perplexity; concatenates the target abstract, the original text and an indicator set and inputs a prompt word into a large language model to score the target abstract to obtain a self-scoring result; weightedly fuses the comprehensive semantic evaluation score, the natural coherence score, the self-scoring result and the Rouge indicator to obtain a quality score of the target abstract, and evaluates the generation quality of the target abstract based on the quality score; and improves the evaluation efficiency of the abstract quality.

Description

Technical Field

[0001] The present invention relates to the technical field of text analysis, and in particular to a method for evaluating the quality of scientific research paper abstract generation and related equipment. Background Art

[0002] The purpose of a paper summary is to compress a long document into a fluent, brief text that preserves the significant information in the original text. With the rapid development of artificial intelligence technology and the application of related technologies represented by pre-trained language models, the model technology of automatic text summarization has been significantly improved. However, compared with the target summary model technology, the research on the evaluation indicators of target summarization lags behind. Today, the mainstream evaluation indicator of target summary is still the Rouge indicator, also commonly known as ROUGE (Recall-Oriented Understudy for Gisting Evaluation), which is called the machine summary evaluation indicator. It is a test indicator for measuring the performance of automatic text summarization and sentence summarization. Although it is widely used, it has gradually shown its limitations.

[0003] The quality evaluation of text summaries is usually done manually, with humans reading the document to be evaluated, summarizing the central idea of ​​the document, and then extracting several sentences to generate a reference text summary. Afterwards, the sentences contained in the summary are judged for similarity with the sentences in the text summary of the document to be evaluated output by the summary generation model to obtain the evaluation result of the text summary. This method has high labor costs, strong subjectivity, low accuracy, and is very time-consuming.

[0004] Research shows that the ROUGE metric is only a rough proxy for summarization quality assessment. The following issues remain unresolved when evaluating whether the target summary text quality meets human requirements:

[0005] First, existing summary evaluation methods cannot capture the subtle differences between the semantics of the target summary text. Mainstream evaluation indicators usually only measure the literal shape overlap similarity between the test summary and the reference summary. For example, common summary evaluation indicators such as ROUGE, BLEU, and METEOR only calculate the N-gram overlap between the candidate and the reference. Because the same meaning can be expressed in different ways, sentences with similar literals can also have opposite semantics. Therefore, the literal matching method only provides a simple and general measurement standard, but it is not enough to capture the subtle differences in the semantics of the target summary.

[0006] Secondly, some studies have shown that mainstream evaluation indicators do not correlate well with human evaluation judgments and are not interpretable. Studies have shown that accurate summary evaluation methods should be consistent with manual evaluation. When humans evaluate the quality of target summaries, they usually conduct comprehensive quality considerations from multiple fine-grained perspectives, such as summary relevance, information richness, redundancy, coherence, and whether it has a good structure. The evaluation indicators of summaries also need to start from the fine-grained indicator dimensions that humans care about and provide interpretable score values.

[0007] Finally, the existing evaluation methods of summary models will inhibit the diversity of target summaries generated by summary models. This is because mainstream summary indicators rely on the comparison between the summary to be tested and a single reference summary, and do not take into account the semantics and nature of the original text itself. However, humans start writing summaries after reading the original text. The process of comparing the reference summary and then scoring the target summary does not exist in real life. Moreover, although the summaries written by readers with different knowledge reserves and different reading purposes are different, they are all correct. It cannot be forced to assume that a summary text that "does not look like" a single reference summary is considered an unqualified summary text. This evaluation method will greatly inhibit the diversity of target summaries generated by summary models. Summary of the invention

[0008] The present invention provides a method for evaluating the quality of abstract generation of scientific research papers and related equipment, the purpose of which is to improve the evaluation efficiency of abstract quality from multiple fine-grained perspectives and taking into account the diversity of abstract generation.

[0009] In order to achieve the above object, the present invention provides a method for evaluating the quality of scientific research paper abstract generation, comprising:

[0010] Step 1, obtain the original text, target abstract and references of the target scientific research paper;

[0011] Step 2: Input the original text, target abstract and references of the target scientific research paper into the SBERT model for bidirectional encoding, and obtain the original text embedding vector, abstract text embedding vector and reference text embedding vector. Extract the pseudo target abstract based on the original text and input it into the SBERT model for encoding to obtain the pseudo abstract text embedding vector.

[0012] Step 3, calculating the original text embedding vector, the summary text embedding vector, the reference text embedding vector and the pseudo summary text embedding vector to obtain a comprehensive semantic evaluation score;

[0013] Step 4: Calculate the word-by-word probability and perplexity of the target summary by defining the word sequence in the target summary and converting it into a word embedding vector, and calculate the natural coherence score based on the word-by-word probability and perplexity. The word-by-word probability is used to characterize the probability of each word appearing in the word sequence, and the perplexity is the exponential inverse of the word-by-word probability, which is used to measure the degree of fit of the model to the data distribution.

[0014] Step 5: Concatenate the target summary, the original text, and the indicator set to obtain prompt words containing the task, where the prompt words are used to describe the information of the task, and input the prompt words into a large language model. In the large language model, the target summary is scored according to the indicator set and the original text to obtain a self-scoring result;

[0015] In step 6, the comprehensive semantic evaluation score, natural coherence score, self-rating result and Rouge index are weightedly integrated to obtain the quality score of the target summary, and the generation quality of the target summary is evaluated based on the quality score.

[0016] Specifically, the SBERT model consists of:

[0017] The input layer is used to separate the input sentences into words or subword units;

[0018] BERT encoder, which is used to encode the separation results output by the input layer and generate a context embedding vector for each word or subword unit;

[0019] The pooling layer is used to perform average pooling operations on the context embedding vectors of words or sub-word units to obtain text embedding vectors.

[0020] More specifically, step 3 includes:

[0021] Calculate the similarity between the original text embedding vector and the summary text embedding vector to obtain a first similarity;

[0022] Calculate the similarity between the pseudo summary text embedding vector and the original text embedding vector to obtain a second similarity;

[0023] Calculate the similarity between the abstract text embedding vector and the reference text embedding vector to obtain a third similarity;

[0024] The original text embedding vector, the abstract text embedding vector, the pseudo-abstract text embedding vector, and the reference text embedding vector are input into the generation probability model for analysis to obtain the topic distribution of the original text, the topic distribution of the target abstract, the topic distribution of the pseudo-target abstract, and the topic distribution of the references;

[0025] Calculate the similarity between the topic distribution of the original text and the topic distribution of the target summary to obtain the first JS divergence;

[0026] Calculate the similarity between the topic distribution of the original text and the topic distribution of the pseudo target summary to obtain the second JS divergence;

[0027] Calculate the similarity between the topic distribution of the target abstract and the topic distribution of the references to obtain the third JS divergence;

[0028] The text feature extraction model is used to extract keywords from the original text embedding vector, the abstract text embedding vector, the reference text embedding vector and the pseudo-abstract text embedding vector to obtain the original text keywords, the target abstract keywords, the reference keywords and the pseudo-target abstract keywords;

[0029] The overlap degree is calculated according to the original text keywords, the target abstract keywords, the reference keywords and the pseudo target abstract keywords to obtain the first overlap degree, the second overlap degree and the third overlap degree. The first overlap degree is the overlap degree between the target abstract and the original text, the second overlap degree is the overlap degree between the pseudo target abstract and the original text, and the third overlap degree is the overlap degree between the target abstract and the reference;

[0030] The information entropy of the first similarity, the second similarity, the third similarity, the first JS divergence, the second JS divergence, the third JS divergence, the first overlap, the second overlap, and the third overlap is calculated respectively, and a comprehensive semantic evaluation score is calculated based on all information entropy values.

[0031] Furthermore, the expression of the comprehensive semantic evaluation score is:

[0032]

[0033] in, represents the information entropy function, represents the first similarity, represents the second similarity, represents the third similarity, represents the first degree of coincidence, represents the second degree of coincidence, represents the third degree of coincidence, represents the first JS divergence, represents the second JS divergence, represents the third JS divergence, represents the original text embedding vector, represents the summary text embedding vector, represents the reference text embedding vector, represents the pseudo-summary text embedding vector, Indicates the original keywords. Indicates the abstract keywords. Indicates the reference keywords. Indicates pseudo target summary keywords.

[0034] More specifically, step 4 includes:

[0035] Define the word sequence in the target summary ;

[0036] The word sequence in the target summary is transformed through the SBERT model Convert to word embedding vector ;

[0037] By formula Calculate the word-by-word probability of the target summary, where represents the joint probability of word sequence, Representing word sequence The words, Indicates the total number of words;

[0038] By formula Calculate the perplexity of the target summary;

[0039] Based on word-by-word probability and perplexity Calculating the natural coherence score , natural coherence score The expression is:

[0040] Among them, represents the weight coefficient.

[0041] More specifically, the set of indicators includes:

[0042] The accuracy, completeness, clarity, logic, ability to generalize, standardization, originality, innovation, length and conciseness of the abstract.

[0043] Furthermore, the expression of the self-scoring result is:

[0044]

[0045] in, Represents a collection of indicators.

[0046] Specifically, the quality score of the target summary The expression is:

[0047]

[0048] in, Corresponding to the comprehensive semantic evaluation score , Natural Coherence Score , self-rating results , The weight coefficient of the indicator.

[0049] The present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a method for evaluating the quality of generating abstracts of scientific research papers when executing the computer program.

[0050] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, a method for evaluating the quality of generating abstracts of scientific research papers is implemented.

[0051] The above scheme of the present invention has the following beneficial effects:

[0052] The present invention obtains the original text, target abstract and references of the target scientific research paper; inputs the original text, target abstract and references of the target scientific research paper into the SBERT model for bidirectional encoding in sequence to obtain the original text embedding vector, the abstract text embedding vector and the reference text embedding vector; extracts a pseudo target abstract based on the original text and inputs it into the SBERT model for encoding to obtain the pseudo abstract text embedding vector; calculates the original text embedding vector, the abstract text embedding vector, the reference text embedding vector and the pseudo abstract text embedding vector to obtain a comprehensive semantic evaluation score; calculates the word-by-word probability and perplexity of the target abstract by defining the word sequence in the target abstract and the SBERT model, and calculates the natural coherence score based on the word-by-word probability and perplexity; connects the target abstract, the original text and the indicator set in series to obtain a score containing The prompt word of the task is input into a large language model, and the target summary is scored in the large language model according to the indicator set and the original text to obtain a self-scoring result; the comprehensive semantic evaluation score, the natural coherence score, the self-scoring result and the Rouge indicator are weightedly fused to obtain the quality score of the target summary, and the generation quality of the target summary is evaluated based on the quality score; compared with the prior art, the present invention uses the SBERT model for bidirectional encoding, and weightedly fuses the comprehensive semantic evaluation score, the natural coherence score and the self-scoring result with the Rouge indicator to form a complete evaluation framework, considers the diversity of summary generation, evaluates the target summary from multiple fine-grained angles such as the comprehensive semantic evaluation score, the natural coherence score and the self-scoring result, and improves the evaluation efficiency of the summary quality.

[0053] Other beneficial effects of the present invention will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a schematic diagram of a flow chart of an embodiment of the present invention;

[0055] Figure 2 Schematic diagram of a terminal device in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, the following will be described in detail with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a locking connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0058] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0059] In view of the existing problems, the present invention provides a method for evaluating the quality of scientific research paper abstract generation and related equipment.

[0060] like Figure 1 As shown, an embodiment of the present invention provides a method for evaluating the quality of scientific research paper abstract generation, comprising:

[0061] Step 1: Obtain the original text, target abstract and references of the target scientific research paper. The original text refers to the main text of the target scientific research paper except the abstract and references, and the target abstract is a machine-generated abstract.

[0062] Step 2: Input the original text, target abstract and references of the target scientific research paper into the SBERT model for bidirectional encoding, and obtain the original text embedding vector, abstract text embedding vector and reference text embedding vector. Extract the pseudo target abstract based on the original text and input it into the SBERT model for encoding to obtain the pseudo abstract text embedding vector.

[0063] Step 3, calculating the original text embedding vector, the summary text embedding vector, the reference text embedding vector and the pseudo summary text embedding vector to obtain a comprehensive semantic evaluation score;

[0064] Step 4: Calculate the word-by-word probability and perplexity of the target summary by defining the word sequence in the target summary and converting it into a word embedding vector, and calculate the natural coherence score based on the word-by-word probability and perplexity. The word-by-word probability is used to characterize the probability of each word appearing in the word sequence given the original text. The perplexity is the exponential inverse of the word-by-word probability, which is used to measure the degree of fit of the model to the data distribution.

[0065] Step 5: Concatenate the target summary, the original text, and the indicator set to obtain a prompt word containing the task. The prompt word is used to describe the information of the task to guide the large language model to solve the problem according to the information, and the prompt word is input into the large language model. The target summary is scored in the large language model according to the indicator set and the original text to obtain a self-scoring result.

[0066] In step 6, the comprehensive semantic evaluation score, natural coherence score, self-rating result and Rouge index are weightedly fused to obtain the quality score of the target summary, and the generation quality of the target summary is evaluated based on the quality score. The Rouge index is used to compare the similarity between the original summary and the pseudo target summary, where the original summary is a summary completed manually.

[0067] In an embodiment of the present invention, an SBERT model is used to obtain an embedding vector of a text. The SBERT model performs bidirectional encoding of the text through a bidirectional encoder representation transformer BERT, and can capture the semantic details of a sentence.

[0068] Most preferably, the SBERT model includes:

[0069] The input layer is used to separate the input sentences into words or subword units. Sentences are separated by special [SEP] tags.

[0070] BERT encoder, which is used to encode the segmentation results output by the input layer and generate the context embedding vector of each word or subword unit; the pre-trained BERT model is used as the encoder. The BERT model consists of multiple layers of Transformer blocks, each of which includes a multi-head attention mechanism and a feedforward neural network. The sentence is encoded through the multi-layer Transformer blocks of the BERT model to generate the context embedding vector of each word;

[0071] The pooling layer is used to perform average pooling operations on the context embedding vectors of words or subword units to obtain text embedding vectors. In order to obtain a fixed-length embedding vector for a sentence, the SBERT model operates on the pooling layer and uses average pooling to take the average value of the context embedding vector of each word as the embedding vector of the sentence.

[0072] The bidirectional encoding process of the SBERT model is the core of the SBERT model, which encodes text in two directions;

[0073] Forward encoding is to encode the input sentence sequence from left to right to generate the forward embedding vector of each word. For example, if the sentence is "the cat is on the mat", the embedding vector generated after forward encoding is " ”;

[0074] Backward encoding is to encode the input sentence sequence from right to left to generate the backward embedding vector of each word. For example, the embedding vector generated by backward encoding for "the cat is on the mat" is " ”;

[0075] The BERT model combines the forward encoding results and the backward encoding results to generate a bidirectional context embedding vector for each word. This bidirectional encoding method enables the embedding vector of each word to capture its contextual information in the sentence, thereby improving the accuracy of semantic representation.

[0076] In the embodiment of the present invention, the original text embedding vector includes the embedding vectors of multiple sentences or words, namely , Represents the number of sentences or words in the original text. The summary text embedding vector includes the embedding vectors of multiple sentences or words, that is, , Represents the number of sentences or words in the abstract. The reference text embedding vector includes the embedding vectors of multiple sentences or words, that is, , Indicates the number of sentences or words in the reference.

[0077] The embodiment of the present invention randomly extracts significant sentences from the original text, concatenates all significant sentences to obtain a pseudo target summary, encodes the pseudo target summary using the SBERT model, and obtains a pseudo summary text embedding vector including the embedding vectors of multiple sentences or words, that is, ,in, Indicates the number of sentences or words in the pseudo target summary. The number of sentences in the pseudo target summary is consistent with the number of sentences in the original text.

[0078] In an embodiment of the present invention, a significant sentence can be defined as a sentence that is close to the topic and appears frequently in the text, that is, a central sentence. According to reading habits, the embodiment of the present invention believes that significant sentences mostly appear in the front and middle parts of the original text. The embodiment of the present invention does not limit the method of extracting significant sentences.

[0079] In the embodiment of the present invention, the functions of the pseudo target summary mainly include:

[0080] Benchmark comparison: Provides an additional benchmark to compare the differences between the target summary and the pseudo-target summary, helping to identify the shortcomings of the target summary;

[0081] Improve evaluation stability: By adding pseudo-target summaries, we can reduce randomness in the evaluation process and improve the stability and reliability of evaluation results.

[0082] Multi-dimensional evaluation: Comprehensively consider the relationship between the generated target summary, pseudo-target summary, original text, and references to provide more comprehensive and multi-dimensional evaluation results.

[0083] Most preferably, step 3 comprises:

[0084] Calculate the similarity between the original text embedding vector and the summary text embedding vector to obtain a first similarity;

[0085] Calculate the similarity between the pseudo summary text embedding vector and the original text embedding vector to obtain a second similarity;

[0086] Calculate the similarity between the abstract text embedding vector and the reference text embedding vector to obtain a third similarity;

[0087] The original text embedding vector, abstract text embedding vector, and reference text embedding vector are input into the generation probability model for analysis to obtain the topic distribution of the original text, the topic distribution of the target abstract, the topic distribution of the pseudo-target abstract, and the topic distribution of the reference. The topic distribution refers to the distribution of text content on multiple potential topics. The topic distribution is usually reflected by the topic model (such as LDA (Latent Dirichlet Allocation)). The information included includes: topic probability, topic feature words, and the number of subjects;

[0088] Calculate the similarity between the topic distribution of the original text and the topic distribution of the target summary to obtain the first JS divergence;

[0089] Calculate the similarity between the topic distribution of the original text and the topic distribution of the pseudo target summary to obtain the second JS divergence;

[0090] Calculate the similarity between the topic distribution of the target abstract and the topic distribution of the references to obtain the third JS divergence;

[0091] The text feature extraction model is used to extract keywords from the original text embedding vector, the abstract text embedding vector, the reference text embedding vector and the pseudo-abstract text embedding vector to obtain the original text keywords, the target abstract keywords, the reference keywords and the pseudo-target abstract keywords;

[0092] Calculate the overlap degree according to the original text keywords, target abstract keywords, reference keywords and pseudo target abstract keywords to obtain the first overlap degree, the second overlap degree and the third overlap degree;

[0093] The information entropy of the first similarity, the second similarity, the third similarity, the first JS divergence, the second JS divergence, the third JS divergence, the first overlap, the second overlap, and the third overlap is calculated respectively, and a comprehensive semantic evaluation score is calculated based on all information entropy values.

[0094] In the embodiment of the present invention, the first similarity, the second similarity, and the third similarity are all semantic similarities, wherein the first similarity is the similarity between the target summary and the original text. For each embedding vector in the target summary, the maximum cosine similarity with all the embedding vectors in the original text is found. The calculation expression of the first similarity is:

[0095]

[0096] in, Indicates summary The The embedding vector of a sentence or word, Indicates original text The The embedding vector of a sentence or word, represents the average function, represents the cosine similarity function;

[0097] The second similarity is the similarity between the pseudo target summary and the original text. The cosine similarity is calculated for each embedding vector in the pseudo target summary and the embedding vector at the corresponding position in the original text, and then the average of these similarities is calculated to obtain the overall similarity score. The calculation expression of the overall similarity score is:

[0098]

[0099] in, Indicates pseudo target summary The The embedding vector of a sentence or word;

[0100] The third similarity is the similarity between the target summary and the reference. For each embedding vector in the target summary, find the maximum cosine similarity with all embedding vectors in the reference, and then average these maximum similarities to get the overall similarity score. The calculation expression of the overall similarity score is:

[0101]

[0102] in, References The The embedding vector of a sentence or word.

[0103] In the embodiment of the present invention, the generative probability model adopted is the LDA model (Latent Dirichlet Allocation), which is used to discover the distribution of potential topics in a document collection. It is assumed that each document is generated by a mixture of multiple topics, and each topic is generated by the distribution of multiple words. Therefore, the embodiment of the present invention applies the LDA model to analyze the topic distribution of the target summary and the original text to obtain the topic probability distribution.

[0104] Specifically, the LDA model structure includes:

[0105] Document-topic distribution :

[0106] Each document has a topic distribution, which represents the probability of each topic appearing in the document. This distribution is usually modeled using Dirichlet distribution.

[0107] Topic-word distribution :

[0108] Each topic has a word distribution, which represents the probability of each word appearing in the topic. This distribution is also modeled using Dirichlet distribution.

[0109] The generation process includes: for each document, extract a topic distribution from the Dirichlet distribution, for each word, extract a topic from the topic distribution of the document, and extract a word from the word distribution of the selected topic;

[0110] The data analysis process of the LDA model is:

[0111] (1) Initialization:

[0112] Set the number of topics K, initialize the document-topic distribution and topic-word distribution;

[0113] (2) Iteration process:

[0114] For each word in each document, reassign topics according to the current distribution and update the document-topic distribution and topic-word distribution;

[0115] (3) Output results:

[0116] Through multiple iterations, converge to a stable document-topic distribution and topic-word distribution, and finally obtain the topic distribution of each document and the word distribution of each topic. In the embodiment of the present invention, the topic distribution obtained by applying the LDA model is the topic distribution of the original text. , the topic distribution of the target summary , topic distribution of pseudo target summary , the topic distribution of references .

[0117] Next, the embodiment of the present invention uses the topic distribution obtained by applying the LDA model to calculate the JS divergence, as follows:

[0118] Calculate the topic distribution similarity between the target summary and the original text to get the first JS divergence If the topic distribution of the abstract is very similar to that of the original text, the jsd value is low, indicating that the topics of the two are similar. The expression is:

[0119]

[0120] Calculate the topic distribution similarity between the pseudo target summary P and the original text O to get the second JS divergence. If the topic distribution of the pseudo target summary is very similar to that of the original text, the jsd value is low, indicating that the two topics are similar. The second JS divergence The expression is:

[0121]

[0122] Calculate the similarity of the topic distribution between the generated target summary S and the reference R to get the third JS divergence. If the topic distribution of the generated target summary is very similar to that of the reference, the jsd value is low, indicating that the topics of the two are similar. The expression of the third JS divergence is:

[0123]

[0124] In the embodiment of the present invention, the text feature extraction model is a TE-IDF model (Term Frequency-Inverse Document Frequency), which is used to measure the importance of a word in a document. It combines the term frequency (TF, TermFrequency) and the inverse document frequency (IDF, Inverse Document Frequency) to evaluate the weight of each word. The specific process is as follows:

[0125] Extract the target summary and the keywords of the original text to get the keywords of the original text , Generate target summary keywords , pseudo target keywords 、References Keywords , keyword weight set ;

[0126] Preprocess the document

[0127] Words: Split the document into individual words or phrases;

[0128] Stop word removal: Remove common words that contribute little to the text content, such as "de", "shi", "zai", etc.;

[0129] Stemming or lemmatization: Reduce words to their basic forms, for example, "running" is reduced to "run";

[0130] Calculate term frequency (TF)

[0131] Count the number of occurrences of each word in the document, and calculate the term frequency of each word according to the following formula:

[0132]

[0133] Calculate inverse document frequency (IDF)

[0134] Count the number of documents containing a specific word, and calculate the IDF value of each word according to the following formula:

[0135]

[0136] Among them, is the total number of documents, represents the number of documents containing the word ;

[0137] Calculate the TF-IDF value

[0138] Calculate the TF-IDF value of each word in the document, and the expression is:

[0139]

[0140] Extract keywords

[0141] Sort the words according to the TF-IDF value of each word, and select several words with the highest TF-IDF values as keywords.

[0142] Calculate the coincidence degree:

[0143] The information coincidence degree is used to represent the keyword similarity between the target summary, pseudo-target summary and the original text, references. A high coincidence degree means that the target summary or pseudo-target summary is closer to the content of the original text or references, thus reflecting the accuracy and relevance of the target summary generation.

[0144] Generate the first coincidence degree, and the first coincidence degree is the coincidence degree between the target summary and the original text, and the expression is:

[0145]

[0146] Generate the second coincidence degree, and the second coincidence degree is the coincidence degree between the pseudo-target summary and the original text, and the expression is:

[0147]

[0148] Generate the third degree of overlap, which is the overlap between the target abstract and the references. The expression is:

[0149]

[0150] Calculate information entropy:

[0151] Entropy is an indicator to measure the uncertainty of random variables. The larger the entropy, the greater the uncertainty and the greater the amount of information. , its information entropy Defined as:

[0152]

[0153] in, is a random variable The value is probability.

[0154] Calculating probability distributions

[0155] For each calculation result, normalize it to get a probability distribution. For example, for the similarity :

[0156]

[0157] in, The similarity value is The number of times is the total number of all similarity values;

[0158] For the first similarity, the information entropy is calculated by the following formula:

[0159]

[0160] Similarly, the entropy values ​​of other sets can also be calculated using the above formula.

[0161] The embodiment of the present invention uses a comprehensive semantic evaluation score to comprehensively evaluate the semantic relevance and quality between the generated target summary and the original text, pseudo target summary, and references; combines different similarities, topic distributions, and entropy values ​​of keyword overlap to measure the multi-dimensional quality of the generated target summary. The expression of the comprehensive semantic evaluation score is:

[0162]

[0163] in, represents the information entropy function, represents the first similarity, represents the second similarity, represents the third similarity, represents the first degree of coincidence, represents the second degree of coincidence, represents the third degree of coincidence, represents the first JS divergence, represents the second JS divergence, represents the third JS divergence, represents the original text embedding vector, represents the summary text embedding vector, represents the reference text embedding vector, represents the pseudo-summary text embedding vector, Indicates the original keywords. Indicates the abstract keywords. Indicates the reference keywords. Indicates pseudo target summary keywords.

[0164] Most preferably, step 4 comprises:

[0165] Define the word sequence in the target summary , ;

[0166] Define the SBERT model as , which can be used for each word in the word sequence In the given previous Calculate the probability under the condition ;

[0167] By formula Calculate the word-by-word probability of the target summary, where represents the joint probability of word sequence, Representing word sequence The words, Indicates the total number of words;

[0168] By formula Calculate the perplexity of the target summary. Perplexity (PP) is the inverse of the word-by-word probability index, which is used to measure the degree of fit of the model to the data distribution. Perplexity provides a standardized measurement indicator for comparing the performance of different models.

[0169] Based on word-by-word probability and perplexity Calculating the natural coherence score , natural coherence score The expression is:

[0170] in, Represents the weight coefficient, which can be adjusted according to actual needs; the natural coherence of the text can be indirectly evaluated by the word-by-word probability; if a language model can assign a higher probability to a natural and coherent text sequence, then the model is considered to perform well in terms of natural coherence.

[0171] Most preferably, the set of indicators includes:

[0172] The target summary should be accurate, complete, clear, logical, generalizable, standard, original, innovative, long, and concise.

[0173] Specifically, the indicator set MetricSet={Accuracy and completeness: whether the target summary accurately summarizes the main ideas, methods, results and conclusions of the original text, and whether it contains necessary background information,

[0174] Clarity and logic: Is the sentence structure of the target abstract clear? Is the logical relationship coherent? Can readers understand the author's ideas and reasoning process?

[0175] Information summarization ability: whether the target abstract can effectively summarize the content of the text, including information on the research background, purpose, methods, results and conclusions,

[0176] Language standardization: whether the language of the target abstract is accurate and standard, whether it conforms to the writing standards of academic papers, whether there are grammatical errors or inappropriate words,

[0177] Originality and innovation: whether the target abstract demonstrates the author's research results and innovative ideas, and whether it has a certain degree of originality and innovation.

[0178] Length and conciseness: Is the target summary of appropriate length? Is it as concise as possible while maintaining the completeness of the information, avoiding redundancy and nonsense?

[0179] Standardized use of terms: whether the terms used in the target abstract are accurate and appropriate, and whether they can clearly convey the core concepts of the paper.

[0180] Specific examples or data: whether specific cases, data or experimental results are provided to support the arguments and conclusions to enhance the credibility and persuasiveness of the target summary.

[0181] Mention of research limitations: Whether the possible limitations of the research are mentioned in the objective summary can reflect the author's in-depth understanding and objective evaluation of the research.

[0182] Research prospects or applications: Whether the potential application prospects of the research or the impact on related fields are briefly described, which will help readers understand the significance and importance of the research.

[0183] The target summary S, the original document O and the indicator set MetricSet are concatenated into a complete prompt word containing a specific task and input into the large language model (LLM). The large language model LLM scores S according to the input MetricSet and O to obtain the self-scoring result. The expression of the self-scoring result is:

[0184]

[0185] in, Represents a collection of indicators.

[0186] In an embodiment of the present invention, the specific task is to let LLM evaluate the quality of the target summary according to the information contained in the prompt words. The prompt words are used to describe the task information. The information included may be the target summary S, the original document O and the indicator set MetricSet, as well as the task description that concatenates these information.

[0187] It should be noted that the large language model is a model composed of a conventional artificial neural network. The embodiment of the present invention does not involve improvements to the structure and data processing method of the model, so its data processing process will not be described in detail.

[0188] In the embodiment of the present invention, the Rouge indicator selects Rouge-L, which is used to evaluate the longest common subsequence (LCS) between the target summary and the reference summary. The specific acquisition process is as follows:

[0189] Given two sequences X and Y, LCS is the longest subsequence that appears in both sequences; for example, and The longest common subsequence of is .

[0190]

[0191] in:

[0192] is the recall rate based on the longest common subsequence, It is the accuracy rate based on the longest common subsequence; It is a parameter that controls the balance between recall and precision and is usually set to 1, meaning that precision and recall are equally important.

[0193] Most preferably, the quality score of the target summary The expression is:

[0194]

[0195] in, Corresponding to the comprehensive semantic evaluation score , Natural Coherence Score , self-rating results , The weight coefficient of the indicator, .

[0196] The embodiment of the present invention assumes that the input original text is a paper titled "Research on Image Target Detection Algorithm Based on Deep Learning", the full text of which is denoted as O, the machine-generated target summary is denoted as S, and the extracted pseudo target summary is denoted as R.

[0197] Step 1: Semantics

[0198] Use the SBERT model to obtain text embedding vectors: Assume that the SBERT embedding of O is: ;

[0199] The SBERT embedding of S is: ;

[0200] The SBERT embedding for R is: ;

[0201] Extract sentences from O to construct a pseudo target summary P. Assume that the SBERT embedding of P is:

[0202]

[0203] The first similarity, second similarity, and third similarity are calculated to be 0.82, 0.91, and 0.75 respectively;

[0204] Calculate the topic distribution JS divergence:

[0205] Assume the topic distribution is:

[0206]

[0207]

[0208]

[0209]

[0210] but:

[0211]

[0212] Calculate keyword overlap:

[0213] Assume the keywords are:

[0214]

[0215]

[0216]

[0217]

[0218] but:

[0219]

[0220]

[0221]

[0222] Calculate partial entropy:

[0223]

[0224]

[0225]

[0226] Calculating Semantics Composite Score ;

[0227] Step 2: Word Probability

[0228] Assume that the word sequence of S is: ;

[0229] The word-by-word probability calculated using SBERT is 0.000025;

[0230] The calculated perplexity is 5.82;

[0231] The calculated WordProbability comprehensive score is 8.21 (taking the logarithm);

[0232] Step 3: LLM Self-Assessment SEMLM

[0233] Construct MetricSet={semantic similarity sim, topic JS divergence jsd, keyword overlap kov, natural coherence WordProbability}

[0234] Input S, O and MetricSet into LLM. Assume that LLM gives a score of 82 points, the full score is 100 points, and the self-rating is 82 points;

[0235] Step 4: Comprehensive Rating

[0236] Assume the ROUGE score is 0.52;

[0237] Weighted fusion to obtain the quality score of the target summary ;

[0238] Score=0.3Semantics+0.2WordProbability+0.3SEMLM+0.2ROUGE

[0239] =0.33.67+0.2(-8.21)+0.382+0.20.52

[0240] =70.54

[0241] In summary, the quality score of the target summary is 70.54.

[0242] The embodiment of the present invention obtains the original text, target abstract and references of the target scientific research paper; inputs the original text, target abstract and references of the target scientific research paper into the SBERT model for bidirectional encoding in sequence to obtain the original text embedding vector, the abstract text embedding vector and the reference text embedding vector; extracts a pseudo target abstract based on the original text and inputs it into the SBERT model for encoding to obtain the pseudo abstract text embedding vector; calculates the original text embedding vector, the abstract text embedding vector, the reference text embedding vector and the pseudo abstract text embedding vector to obtain a comprehensive semantic evaluation score; calculates the word-by-word probability and perplexity of the target abstract by defining the word sequence in the target abstract and the SBERT model, and calculates the natural coherence score based on the word-by-word probability and perplexity; concatenates the target abstract, the original text and the indicator set to obtain a score containing any The prompt words of the service are input into a large language model, and the target summary is scored in the large language model according to the indicator set and the original text to obtain a self-scoring result; the comprehensive semantic evaluation score, the natural coherence score, the self-scoring result and the Rouge indicator are weightedly fused to obtain the quality score of the target summary, and the generation quality of the target summary is evaluated based on the quality score; compared with the prior art, the embodiment of the present invention performs bidirectional encoding through the SBERT model, and weightedly fuses the comprehensive semantic evaluation score, the natural coherence score, the self-scoring result and its summary evaluation indicators from three aspects to form a complete evaluation framework, considers the diversity of summary generation, evaluates the generated summary from multiple fine-grained angles such as the comprehensive semantic evaluation score, the natural coherence score and the self-scoring result, and improves the evaluation efficiency of the summary quality.

[0243] An embodiment of the present invention also provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a method for evaluating the quality of generating abstracts of scientific research papers when executing the computer program.

[0244] like Figure 2As shown, the terminal device D10 provided in the embodiment of the present invention includes: at least one processor D100 ( Figure 2 Only one processor is shown in the figure), a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100, wherein the processor D100 implements the steps in any of the above-mentioned method embodiments when executing the computer program D102. Alternatively, the processor D100 implements the functions of each module / unit in the above-mentioned device embodiments when executing the computer program D102.

[0245] The processor may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0246] In some embodiments, the memory may be an internal storage unit of the terminal device, such as a hard disk or memory of the terminal device. In other embodiments, the memory may also be an external storage device of the terminal device, such as a plug-in hard disk, a smart memory card (SMC, Smart Media Card), a secure digital (SD, Secure Digital) card, a flash card (Flash Card), etc. equipped on the terminal device. Further, the memory may also include both an internal storage unit of the terminal device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (Boot Loader), data and other programs, such as the program code of the computer program, etc. The memory may also be used to temporarily store data that has been output or is to be output.

[0247] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0248] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0249] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, a method for evaluating the quality of generating abstracts of scientific research papers is implemented.

[0250] If the integrated module is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above method embodiments, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the construction device / terminal device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. For example, USB flash drive, mobile hard disk, disk or optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0251] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for evaluating the quality of scientific research paper abstract generation, characterized in that: include: Step 1, obtain the original text, target abstract and references of the target scientific research paper; Step 2: The original text, target abstract and references of the target scientific research paper are sequentially input into the SBERT model for bidirectional encoding to obtain the original text embedding vector, the abstract text embedding vector and the reference text embedding vector; a pseudo target abstract is extracted based on the original text and input into the SBERT model for encoding to obtain the pseudo abstract text embedding vector; Step 3, calculating the original text embedding vector, the summary text embedding vector, the reference text embedding vector and the pseudo summary text embedding vector to obtain a comprehensive semantic evaluation score; Step 4, calculating the word-by-word probability and perplexity of the target summary by defining the word sequence in the target summary and converting it into a word embedding vector, and calculating the natural coherence score based on the word-by-word probability and the perplexity, wherein the word-by-word probability is used to characterize the probability of each word appearing in the word sequence, and the perplexity is the exponential inverse of the word-by-word probability, which is used to measure the degree of fit of the model to the data distribution; Step 5, concatenating the target summary, the original text and the indicator set to obtain prompt words containing the task, the prompt words are used to describe the information of the task, and the prompt words are input into a large language model, and the target summary is scored in the large language model according to the indicator set and the original text to obtain a self-scoring result; Step 6: Weightedly fuse the comprehensive semantic evaluation score, the natural coherence score, the self-scoring result and the Rouge indicator to obtain a quality score of the target summary, and evaluate the generation quality of the target summary based on the quality score. The Rouge indicator is used to compare the similarity between the summary and the pseudo target summary.

2. The method for evaluating the quality of scientific research paper abstract generation according to claim 1, characterized in that: The SBERT model includes: The input layer is used to separate the input sentences into words or subword units; A BERT encoder is used to encode the separation result output by the input layer to generate a context embedding vector for each word or subword unit; The pooling layer is used to perform an average pooling operation on the context embedding vector of the word or sub-word unit to obtain a text embedding vector.

3. The method for evaluating the quality of scientific research paper abstract generation according to claim 2, characterized in that: The step 3 comprises: Calculating the similarity between the original text embedding vector and the summary text embedding vector to obtain a first similarity; Calculating the similarity between the pseudo summary text embedding vector and the original text embedding vector to obtain a second similarity; Calculating the similarity between the abstract text embedding vector and the reference text embedding vector to obtain a third similarity; Input the original text embedding vector, the summary text embedding vector, the pseudo summary text embedding vector, and the reference document text embedding vector into a generation probability model for analysis to obtain the topic distribution of the original text, the topic distribution of the target summary, the topic distribution of the pseudo target summary, and the topic distribution of the reference document; Calculate the similarity between the topic distribution of the original text and the topic distribution of the target summary to obtain a first JS divergence; Calculate the similarity between the topic distribution of the original text and the topic distribution of the pseudo target summary to obtain a second JS divergence; Calculate the similarity between the topic distribution of the target abstract and the topic distribution of the reference to obtain a third JS divergence; Using a text feature extraction model, extracting keywords from the original text embedding vector, the summary text embedding vector, the reference document text embedding vector, and the pseudo summary text embedding vector to obtain original text keywords, target summary keywords, reference document keywords, and pseudo target summary keywords; Calculating the overlap degree according to the original text keywords, the abstract keywords, the reference keywords and the pseudo target abstract keywords to obtain a first overlap degree, a second overlap degree and a third overlap degree, wherein the first overlap degree is the overlap degree between the target abstract and the original text, the second overlap degree is the overlap degree between the pseudo target abstract and the original text, and the third overlap degree is the overlap degree between the target abstract and the reference; The information entropies of the first similarity, the second similarity, the third similarity, the first JS divergence, the second JS divergence, the third JS divergence, the first overlap, the second overlap, and the third overlap are calculated respectively, and a comprehensive semantic evaluation score is calculated based on all information entropy values.

4. The method for evaluating the quality of scientific research paper abstract generation according to claim 3, characterized in that: The expression of the comprehensive semantic evaluation score is: in, represents the information entropy function, represents the first similarity, represents the second similarity, represents the third similarity, represents the first degree of coincidence, represents the second degree of coincidence, represents the third degree of coincidence, represents the first JS divergence, represents the second JS divergence, represents the third JS divergence, represents the original text embedding vector, represents the summary text embedding vector, represents the reference text embedding vector, represents the pseudo-summary text embedding vector, Indicates the original keywords. Indicates the abstract keywords. Indicates the reference keywords. Indicates pseudo target summary keywords.

5. The method for evaluating the quality of scientific research paper abstract generation according to claim 4, characterized in that: The step 4 comprises: Define the sequence of words in the target summary ; The word sequence in the target summary is transformed into Convert to word embedding vector ; By formula Calculate the word-by-word probability of the target summary, where represents the joint probability of word sequence, Representing word sequence The words, Indicates the total number of words; By formula Calculating the perplexity of the target summary; Based on the word-by-word probability and the perplexity Calculating the natural coherence score , the natural coherence score The expression is: in, Represents the weight coefficient.

6. The method for evaluating the quality of scientific research paper abstract generation according to claim 5, characterized in that: The indicator set includes: The accuracy, completeness, clarity, logic, ability to generalize, standardization, originality, innovation, length and conciseness of the abstract.

7. The method for evaluating the quality of scientific research paper abstract generation according to claim 6, characterized in that: The expression of the self-scoring result is: in, Represents a collection of indicators.

8. The method for evaluating the quality of scientific research paper abstract generation according to claim 7, characterized in that: The quality score of the summary The expression is: in, Corresponding to the comprehensive semantic evaluation score , Natural Coherence Score , self-rating results , The weight coefficient of the indicator.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for evaluating the quality of scientific research paper abstract generation as described in any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the method for evaluating the quality of scientific research paper abstract generation as described in any one of claims 1 to 8.

Images