Paper innovation quality measurement index generation method based on large language model

Abstract

The invention provides a method for generating a measurement index of thesis innovation quality based on a big language model, and relates to the technical field of thesis innovation quality measurement, and the method comprises the steps: firstly obtaining to-be-tested thesis data; the to-be-tested paper data is input into a preset science and technology paper innovation quality measurement model, the science and technology paper innovation quality measurement model comprises a novelty measurement module, a usefulness score calculation module and a measurement index calculation module, and the novelty measurement module carries out keyword extraction on the to-be-tested paper data through a constructed large language model; calculating to obtain a novelty score according to the keyword; the usefulness score calculation module calculates a usefulness score according to the to-be-tested paper data; and inputting the novelty score and the usefulness score into a measurement index calculation module, wherein a measurement index is used for measuring the innovation quality of the to-be-measured paper. According to the method, the distinction degree of innovative measurement of the paper is effectively improved, the comprehensiveness of prediction dimensions is ensured, measurement index generation is performed on the paper from an objective angle, and the reliability of a measurement result is ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of computer intelligent algorithms, and in particular to a method for generating measurement indicators for the innovation quality of papers based on a large language model. Background Art

[0002] As the core carrier for displaying and disseminating scientific research results, the innovation measurement of scientific and technological papers constitutes an indispensable foundation of the scientific and technological measurement system. In the innovation measurement methodology, it is mainly divided into subjective prediction methods of paper content and objective prediction methods based on objective measurement indicators. Since the subjective prediction method is overly influenced by human factors, the measurement results of the innovative quality of the paper are not stable and reliable. However, with the development of artificial intelligence technology, the combination of artificial intelligence models and objective measurement indicators is used to measure scientific and technological papers, making the measurement results more stable and reliable.

[0003] At present, the use of objective measurement indicators and artificial intelligence models for paper measurement has received widespread attention. For example, objective measurement indicators such as the number of references, citation frequency, and reference combination based on the citation perspective have been introduced into artificial intelligence models for scientific paper measurement, which greatly enriches the dimension and accuracy of the measurement system. The prior art discloses an innovative measurement method for papers based on the SAO-ADV structure. It uses an improved SAO-ADV model to fully extract the content of the paper and connect the subject words of the paper with a complete grammatical structure to make the prediction results more comprehensive. At the same time, the semantic similarity model can be used to directly predict the content of the paper without the help of external measurement indicators such as citations, which can better reflect the quality of the content of the paper. However, the prediction of scientific papers based on indicators such as word combination co-occurrence and topic novelty from a text perspective is difficult to predict. In the research on the prediction dimension of scientific papers, it belongs to a single dimension. The single-dimensional prediction method easily leads to the problems of low discrimination of the measurement indicators of paper innovation and incomplete prediction dimensions. Summary of the invention

[0004] In order to solve the problems of low discrimination, incomplete prediction dimensions and unreliable results in the above-mentioned prior art paper measurement, the present invention proposes a method for generating measurement indicators for the innovation quality of papers based on a large language model, which effectively improves the discrimination of the paper innovation measurement, ensures the comprehensiveness of the prediction dimensions, and generates measurement indicators for the paper from an objective perspective to ensure the reliability of the measurement results.

[0005] In order to achieve the above technical effects, the technical solution of the present invention is as follows: A method for generating measurement indicators of paper innovation quality based on a large language model includes the following steps: S1. Obtain the paper data to be tested; S2. Input the paper data to be tested into a preset scientific paper innovation quality measurement model, wherein the scientific paper innovation quality measurement model includes a novelty measurement module, a usefulness score calculation module and a measurement index calculation module, and the novelty measurement module and the usefulness score calculation module are connected to the measurement index calculation module in parallel; S3. constructing a large language model using the novelty measurement module, extracting keywords from the paper data to be tested through the large language model, and calculating the novelty score of the paper to be tested based on the keywords; the usefulness score calculation module calculates the usefulness score of the paper to be tested based on the paper data to be tested; S4. Input the novelty score and the usefulness score into the measurement index calculation module, calculate and output the measurement index of the paper to be tested, and the measurement index is used to measure the innovation quality of the paper to be tested.

[0006] Preferably, the paper data to be tested includes a text dataset of the paper title and abstract to be tested.

[0007] Preferably, the novelty measurement module includes an automatic keyword extraction submodule and a novelty score calculation submodule that are interconnected. The automatic keyword extraction submodule is used to construct a large language model, and to extract keywords from the test paper data through the large language model; the novelty score calculation submodule is used to calculate the novelty score of the test paper based on the keywords.

[0008] Preferably, the large language model is a word vector generation model and a keyword recognition model connected to each other, and the keyword extraction of the test paper data by the large language model includes: S301. Input the test paper data into the word vector generation model for word segmentation to obtain multiple word segments, map each word segment into a high-dimensional vector space and convert it into a word vector to obtain a word vector set; S302. Taking the word vector set as an input sequence, using the keyword recognition model to capture complex patterns and long-distance dependencies in the input sequence, and obtaining a keyword set; S303. Take the keyword set as a high-dimensional vector, convert the high-dimensional vector into a dimensional vector required for the target task, use an activation function to convert the dimensional vector into a probability distribution of keywords, and divide the keywords into problem words and method words according to the probability distribution.

[0009] Preferably, the novelty score of the paper to be tested is calculated based on the keywords, including: S311. Take the publication year of the paper to be tested as the reference point, compare the problem words and method words in other papers published earlier than the paper to be tested in the same field, calculate the frequency of problem words and method words, and merge the frequency of problem words and method words into the frequency of keyword appearance. ; S312. Based on the frequency of keyword occurrence , the expression for calculating the novelty score is as follows:

[0010] in, represents the novelty score of the paper to be tested, represents the keyword set of the paper to be tested, Representing a collection The number of elements of It represents the kth word that appears in the paper before it is published.

[0011] Preferably, the usefulness score of the paper to be tested is calculated based on the paper data to be tested, including: S321. Divide the citation frequency of the paper data to be tested into intervals and perform score processing to obtain the citation frequency score CPi; S322. Integrate and count the journals of the citing documents of the papers to be tested, and calculate the authoritative citation ratio of each citing document , and assign the authoritative citation ratio within the corresponding sub-area Citing article quality score ; S323. Combine the CPI score with the quality score of the citing literature Multiply them together to get the usefulness score of the paper to be tested. .

[0012] Preferably, the measurement index of the paper to be tested is calculated and outputted using the approximate ideal solution sorting method, including: S401. Based on the novelty score and the usefulness score, construct an initial judgment matrix ; S402. The initial judgment matrix Perform standardization to obtain a standardized matrix Z; S403. Determine the positive ideal solution of the novelty score and usefulness score in the normalized matrix Z and negative ideal solution ; S404. Based on the positive ideal solution and negative ideal solution , and the weighted Euclidean distance is calculated to obtain the distance from each measurement object to the positive ideal solution The positive ideal solution distance and negative ideal solution distance ; S405. Based on the positive ideal solution distance and negative ideal solution distance , calculate the distance to the positive ideal solution and negative ideal solution distance The relative proximity score ; S406. Score the relative proximity Define as the measurement indicators, arrange the measurement indicators, and complete the measurement of the innovation quality of the paper to be measured.

[0013] Preferably, the initial judgment matrix The form is as follows:

[0014] in, Represents the initial judgment matrix The i Line m Column elements, i represents the number of measurement objects, m= 2 represents the number of measurement indicators, represents the novelty score of the paper to be tested, Represents the usefulness score of the paper to be tested.

[0015] Preferably, the initial judgment matrix After standardization, the calculation expression of the standardized matrix Z is as follows:

[0016] After standardization, the standardized matrix Z is as follows:

[0017] in, represents the first i Line m Column elements.

[0018] The present invention also proposes a measurement index generation system for the innovation quality of papers based on a large language model, which is characterized by comprising: The acquisition module is used to obtain the data of the paper to be tested; An input module, used for inputting the paper data to be tested into a preset scientific paper innovation quality measurement model, wherein the scientific paper innovation quality measurement model includes a novelty measurement module, a usefulness score calculation module and a measurement index calculation module, and the novelty measurement module and the usefulness score calculation module are connected to the measurement index calculation module in parallel; The measurement module is used to construct a large language model using the novelty measurement module, extract keywords from the paper data to be tested through the large language model, and calculate the novelty score of the paper to be tested based on the keywords; the usefulness score calculation module calculates the usefulness score of the paper to be tested based on the paper data to be tested; the novelty score and the usefulness score are input into the measurement index calculation module, and the measurement index calculation module calculates and outputs the measurement index of the paper to be tested, and the measurement index is used to measure the innovation quality of the paper to be tested.

[0019] Compared with the prior art, the technical solution of the present invention has the following beneficial effects: The present invention proposes a method for generating measurement indicators of innovative quality of scientific and technological papers based on a large language model. Firstly, a large language model is constructed through a novelty measurement module in the innovative quality measurement model of scientific and technological papers, and then keywords are extracted from the data of the papers to be tested through the large language model. The purpose is to apply the automatic keyword extraction technology of the large language model to the innovative quality measurement of scientific and technological papers, improve the traditional text keyword extraction technology, and enhance the effectiveness of novel keyword extraction. Secondly, the novelty measurement module in the innovative quality measurement model of scientific and technological papers starts from the novelty score dimension of the papers to be tested, and the usefulness score calculation module starts from the usefulness score dimension of the papers to be tested, and inputs the novelty score and the usefulness score into the measurement indicator calculation module for calculation. The measurement indicator calculation module integrates the two dimensions of the novelty score and the usefulness score, and outputs the measurement indicator of the papers to be tested. The measurement indicator is used to measure the innovative quality of the papers to be tested, which effectively improves the discrimination of the innovativeness measurement of the papers, ensures the comprehensiveness of the prediction dimension, and generates measurement indicators for the papers from an objective perspective, so as to ensure the reliability of the measurement results. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic diagram showing a flow chart of a method for generating a measurement index of the innovation quality of a paper based on a large language model proposed in an embodiment of the present invention; Figure 2 A measurement framework diagram showing a paper creation quality measurement model proposed in an embodiment of the present invention; Figure 3 A schematic diagram showing the design principle of automatic keyword extraction using a large language model proposed in an embodiment of the present invention; Figure 4A structural diagram showing a measurement index generation system for the innovation quality of a paper based on a large language model proposed in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The accompanying drawings are only used for illustrative purposes and are not to be construed as limiting the present invention; It is understandable to those skilled in the art that some well-known contents may be omitted in the drawings; The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0022] Example 1 like Figure 1 and Figure 2 As shown, the present invention proposes a method for generating a measurement index of the innovation quality of a paper based on a large language model, comprising the following steps: S1. Obtain the paper data to be tested; In S1, the paper data to be tested includes a data set of titles and abstract texts of the paper to be tested.

[0023] S2. Input the paper data to be tested into a preset scientific paper innovation quality measurement model, wherein the scientific paper innovation quality measurement model includes a novelty measurement module, a usefulness score calculation module and a measurement index calculation module, and the novelty measurement module and the usefulness score calculation module are connected to the measurement index calculation module in parallel; S3. constructing a large language model using the novelty measurement module, extracting keywords from the paper data to be tested through the large language model, and calculating the novelty score of the paper to be tested based on the keywords; the usefulness score calculation module calculates the usefulness score of the paper to be tested based on the paper data to be tested; In S3, the novelty measurement module includes an automatic keyword extraction submodule and a novelty score calculation submodule that are interconnected. The automatic keyword extraction submodule is used to construct a large language model, and the large language model is used to extract keywords from the paper data to be tested; the novelty score calculation submodule is used to calculate the novelty score of the paper to be tested based on the keywords.

[0024] In the automatic keyword extraction submodule, the large language model is a word vector generation model and a keyword recognition model that are connected to each other. In this embodiment, the word vector generation model is selected as the Byte Pair Encoding model (BPE model), and the BPE model is used for accurate word segmentation. The keyword recognition model is selected as the Phi3 model. The Phi3 model is used as a processing tool for the encoder layer in keyword extraction. The large language model fuses the BPE model and the Phi3 model, that is, the large language model is specifically a Phi3-BPE fusion model. See also Figure 3 , extracting keywords from the paper data to be tested by using the large language model, including: S301. Input the test paper data into the word vector generation model for word segmentation to obtain multiple word segments, map each word segment into a high-dimensional vector space and convert it into a word vector to obtain a word vector set; In S301, the paper data to be tested includes a text dataset of the titles and abstracts of the papers to be tested in a specific field {W1, W2...Wn}; the word vector generation model is a BPE model, the BPE model is trained, the text dataset of the titles and abstracts of the papers to be tested {W1, W2...Wn} are input into the BPE model for word segmentation, and each word is mapped to a high-dimensional vector space and converted into a dense vector through the BPE model, the dense vector is the word vector, and the position code is generated to determine the position information, and the word vector set {B1, B2...Bn} is obtained; S302. Taking the word vector set as an input sequence, using the keyword recognition model to capture complex patterns and long-distance dependencies in the input sequence, and obtaining a keyword set; In 302, the keyword recognition model is a Phi3 model with a multi-head self-attention mechanism. The keywords are further processed using the Phi3 model, where the self-attention mechanism calculates the relevance of each word to other words and generates weighted representations. The multi-head mechanism repeats this process multiple times. These weighted representations are then processed through a two-layer fully connected feedforward neural network, with multiple encoder layers stacked together, enabling the Phi3 model to capture complex patterns and long-distance dependencies in the input sequence. In the Phi3 model, multiple encoders are stacked to extract text keywords simultaneously. Each encoder layer is independent but has the same structure. By stacking encoders, the Phi3 model can capture complex patterns and long-distance dependencies in the input sequence and obtain a keyword set {Z1, Z2...Zn} with more detailed word feature representations. S303. Take the keyword set as a high-dimensional vector, convert the high-dimensional vector into a dimensional vector required for the target task, use an activation function to convert the dimensional vector into a probability distribution of keywords, and divide the keywords into problem words and method words according to the probability distribution.

[0025] In S303, the keyword set output by the stacked encoder is linearly transformed as a high-dimensional vector, the high-dimensional vector representation is converted into a dimensional vector required by the target task, and then the Soft-max activation function is applied to generate a probability distribution of the keywords, and the keywords are divided into question words or method words; In the novelty score calculation submodule, the novelty score of the paper to be tested is calculated according to the keywords, including: S311. Take the publication year of the paper to be tested as the reference point, compare the problem words and method words in other papers published earlier than the paper to be tested in the same field, calculate the frequency of problem words and method words, and merge the frequency of problem words and method words into the frequency of keyword appearance. ; S312. Based on the frequency of keyword occurrence , the expression for calculating the novelty score is as follows:

[0026] in, represents the novelty score of the paper to be tested, represents the keyword set of the paper to be tested, Representing a collection The number of elements of It represents the kth word that appears in the paper before it is published.

[0027] In the usefulness score calculation module, the usefulness score of the paper to be tested is calculated based on the paper data to be tested, including: S321. Divide the citation frequency of the paper data to be tested into intervals and perform score processing to obtain the citation frequency score CPi; In S321, the test paper data also includes dividing the test paper data of the selected discipline into Top0.1%, Top0.5%, Top1%, Top5%, Top10%, Top25%, Top50% and the remaining 50% after the test paper data are processed in descending order according to the citation frequency. According to this proportion, papers with zero citation frequency are excluded, and the sample is named "Divided by Zero Citation Frequency". The interval of "Divided by Zero Citation Frequency" is between (0, 0.1%], (0.1%, 0.5%], ( 0.5%, 1.0%], (1.0%, 5.0%], (5.0%, 10.0%], (10.0%, 25.0%], (25.0%, 50.0%] and (50.0%, 100.0%] are assigned 1.6, 1.4, 1.2, 1.0, 0.8, 0.6, 0.4 and 0.2 points respectively. Papers with zero citation frequency are all assigned 0.1 points. Papers with the same citation frequency must be assigned the same score. The citation frequency score CPi of each paper is obtained. The scoring rules are shown in the formula as follows:

[0028] Among them, CPi represents the iThe citation frequency score of the paper data to be tested, "divided by zero citation frequency" means the citation value whose citation frequency is greater than zero; it should be specially stated that in the actual processing process, the scoring rule is not limited to the above rules, and the scoring rule in this embodiment is only used to illustrate an embodiment; S322. Integrate and count the journals of the citing documents of the papers to be tested, and calculate the authoritative citation ratio of each citing document , and assign the authoritative citation ratio within the corresponding sub-area Citing article quality score ; S322 is the citation quality measurement of papers, which measures the proportion of authoritative citations. The more citing documents of the papers to be tested, the more they are collected by high-quality journals, the more useful the papers to be tested are. The definition of authoritative papers is that "the journals to which the citing documents belong are in the Q1 or Q2 zone of the JCR ranking of their fields", and the authoritative citation ratio is calculated. The interval is assigned points; The formula for calculating the authoritative citation ratio is as follows:

[0029] Among them, AC represents the authoritative citation ratio, Indicates i The number of citations of the tested paper by high-quality papers, C represents the i The citation frequency of the papers to be tested; The authoritative citation ratio The scoring rules for interval scoring are as follows:

[0030] Among them, HPi represents the i The quality score of the citing literature of the tested paper.

[0031] S323. Combine the CPI score with the quality score of the citing literature Multiply them together to get the usefulness score of the paper to be tested. , usefulness score The calculation expression is as follows:

[0032] It should be noted that S3 combines the automatic supervised text training mechanism of the Phi3 model with the more accurate word segmentation effect of the BPE model to construct the Phi3-BPE fusion model. For the first time, the automatic keyword extraction technology of the large language model is applied to the measurement of the innovative quality of scientific and technological papers. It combines the two dimensions of novelty from the text perspective and usefulness from the citation perspective, and combines the measurement index calculation module to build a scientific and technological paper innovation quality measurement model, in order to improve the deficiencies of previous research and propose a more comprehensive, accurate and highly discriminative measurement model. Secondly, a scientific and technological paper innovation quality measurement model that comprehensively considers the novelty and usefulness of the paper is proposed. At the same time, the Phi3-BPE fusion model is innovatively applied to improve the traditional text keyword extraction technology and enhance the effectiveness of novel keyword extraction.

[0033] S4. Input the novelty score and the usefulness score into the measurement index calculation module, calculate and output the measurement index of the paper to be tested, and the measurement index is used to measure the innovation quality of the paper to be tested; wherein the measurement index is the innovation quality score; In this embodiment, firstly, a large language model is constructed through the novelty measurement module in the scientific and technological paper innovation quality measurement model, and then the large language model is used to extract keywords from the paper data to be tested, with the purpose of applying the large language model automatic keyword extraction technology to the scientific and technological paper innovation quality measurement, improving the traditional text keyword extraction technology, and enhancing the effectiveness of novelty keyword extraction; secondly, the novelty measurement module in the scientific and technological paper innovation quality measurement model starts from the novelty score dimension of the paper to be tested, and the usefulness score calculation module starts from the usefulness score dimension of the paper to be tested, and inputs the novelty score and the usefulness score into the measurement index calculation module for calculation. The measurement index calculation module integrates the two dimensions of novelty score and usefulness score, and outputs the measurement index of the paper to be tested. The measurement index is used to measure the innovation quality of the paper to be tested, effectively improving the discrimination of the paper innovation prediction, and ensuring the comprehensiveness of the prediction dimension.

[0034] Example 2 This embodiment further explains S4, in order to combine the novelty score and the usefulness score into a comprehensive measurement index, see Figure 2 , the measurement index of the paper to be tested is calculated and output using the approximate ideal solution ranking method (i.e. TOPSIS method). TOPSIS method is a multi-index decision analysis method, which is suitable for measuring papers by integrating multi-dimensional indicators, including: S401. Based on the novelty score and the usefulness score, construct an initial judgment matrix ; In S401, the initial judgment matrix The form is as follows:

[0035] in, Represents the initial judgment matrix The i Line m Column elements, i represents the number of measurement objects, m= 2 indicates the number of measurement indicators, including novelty score and usefulness score; represents the novelty score of the paper to be tested, Represents the usefulness score of the paper to be tested.

[0036] S402. The initial judgment matrix Perform standardization to obtain a standardized matrix Z; In S402, when the TOPSIS method is used, the calculation method of the novelty score and the usefulness score both presents the characteristic that the larger the value, the better. Both are extremely large indicators and do not need to be processed positively. Due to the problem that the values ​​of different data are different, the initial judgment matrix After standardization, the calculation expression of the standardized matrix Z is as follows:

[0037] After standardization, the standardized matrix Z is as follows:

[0038] in, represents the element in the i-th row and m-th column of the standardized matrix Z, the first column { } represents the novelty score of the paper after normalization. The second column { } represents the usefulness score of the tested paper after standardized processing.

[0039] S403. Determine the positive ideal solution of the novelty score and usefulness score in the normalized matrix Z and negative ideal solution ; S404. Based on the positive ideal solution and negative ideal solution , and the weighted Euclidean distance is calculated to obtain the distance from each measurement object to the positive ideal solution The positive ideal solution distance and negative ideal solution distance ; and The specific calculation formulas are as follows: 2 2 S405. Based on the positive ideal solution distance and negative ideal solution distance , calculate the distance to the positive ideal solution and negative ideal solution distance The relative proximity score ; The calculation expression is as follows:

[0040] S406. Score the relative proximity Defined as the measurement index, the measurement index is arranged in descending order to obtain a comprehensive evaluation result of the innovation quality of scientific and technological papers in this field, so as to complete the measurement of the innovation quality of the paper to be tested.

[0041] Example 3 This embodiment further experimentally verifies a method for generating a measurement index of the innovation quality of a paper based on a large language model proposed in the above embodiment; 1) Data selection Papers published in 2018 and 2019 in the field of computer science were selected as samples to be tested, and data from 1965 to 2017 in the same field in Web of Science were crawled for comparison data sets of samples. The authoritative citation data in the usefulness dimension is based on the JCR journal partition published in 2024. After preliminary retrieval, 14,879 sample documents to be tested were obtained, and the comparison data totaled 133,055. After data cleaning, a total of 10,012 papers to be tested were finally obtained.

[0042] 2) Experimental procedures Step 1: Calculate the novelty score, including: Input the title and abstract of the paper to be tested, use the BPE model to segment and vectorize the text, and then input the processed text into the Phi3 model to issue a text keyword extraction instruction; The Phi3 model identifies the problem words and method words of the paper to be tested according to the instructions, and outputs the keyword extraction results of the sample after comparing them with the BPE word segmentation results. The extracted results are summarized in an Excel table, and the titles and abstracts of 10,012 samples are repeatedly substituted into the Phi3 model for keyword extraction using Python loops. After summarizing the extracted problem words and method words, they are compared with the full database data and the frequency of occurrence is calculated. The total number of occurrences is substituted into the calculation formula to obtain the novelty score of the paper.

[0043] Step 2: Calculate the usefulness score, including: divide the citation frequency of the paper to be tested into intervals and assign scores, recorded as Pi, integrate and count the journal partitions to which the citing documents of the paper to be tested belong, calculate the "authoritative citation ratio" of each document and assign corresponding interval scores, recorded as Hi. Multiply the citation frequency score and the citing document quality score to obtain the usefulness score of the paper to be tested.

[0044] Step 3: Calculate the measurement indicators, including: substituting the novelty score and usefulness score calculated above into the TOPSIS model to obtain the paper measurement indicators, and arranging the total scores in descending order to obtain the ranking results of scientific papers in the computer field in 2018 and 2019 in this model measurement.

[0045] Example 4 This embodiment proposes a measurement index generation system for the innovation quality of papers based on a large language model, see Figure 4 ,include: The acquisition module is used to obtain the data of the paper to be tested; An input module, used for inputting the paper data to be tested into a preset scientific paper innovation quality measurement model, wherein the scientific paper innovation quality measurement model includes a novelty measurement module, a usefulness score calculation module and a measurement index calculation module, and the novelty measurement module and the usefulness score calculation module are connected to the measurement index calculation module in parallel; The measurement module is used to construct a large language model using the novelty measurement module, extract keywords from the paper data to be tested through the large language model, and calculate the novelty score of the paper to be tested based on the keywords; the usefulness score calculation module calculates the usefulness score of the paper to be tested based on the paper data to be tested; the novelty score and the usefulness score are input into the measurement index calculation module, and the measurement index calculation module calculates and outputs the measurement index of the paper to be tested, and the measurement indexes are arranged to complete the measurement of the innovation quality of the paper to be tested.

[0046] In this embodiment, firstly, a large language model is constructed through the novelty measurement module in the scientific and technological paper innovation quality measurement model, and then the large language model is used to extract keywords from the paper data to be tested, with the purpose of applying the large language model automatic keyword extraction technology to the scientific and technological paper innovation quality measurement, improving the traditional text keyword extraction technology, and enhancing the effectiveness of novelty keyword extraction; secondly, the novelty measurement module in the scientific and technological paper innovation quality measurement model starts from the novelty score dimension of the paper to be tested, and the usefulness score calculation module starts from the usefulness score dimension of the paper to be tested, and inputs the novelty score and the usefulness score into the measurement index calculation module for calculation. The measurement index calculation module integrates the two dimensions of novelty score and usefulness score, and outputs the measurement index of the paper to be tested. The measurement index is used to measure the innovation quality of the paper to be tested, which effectively improves the discrimination of the paper innovation measurement, ensures the comprehensiveness of the prediction dimension, and generates measurement indicators for the paper from an objective perspective to ensure the reliability of the measurement results.

[0047] Obviously, the above embodiments of the present invention are only examples for clearly illustrating the present invention, and are not intended to limit the implementation methods of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all implementation methods here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A method for generating measurement indicators for the innovation quality of papers based on a large language model, characterized in that: The following steps are involved: S1. Obtain the paper data to be tested; S2. Input the paper data to be tested into a preset scientific paper innovation quality measurement model, wherein the scientific paper innovation quality measurement model includes a novelty measurement module, a usefulness score calculation module and a measurement index calculation module, and the novelty measurement module and the usefulness score calculation module are connected to the measurement index calculation module in parallel; S3. constructing a large language model using the novelty measurement module, extracting keywords from the paper data to be tested through the large language model, and calculating the novelty score of the paper to be tested based on the keywords; the usefulness score calculation module calculates the usefulness score of the paper to be tested based on the paper data to be tested; S4. Input the novelty score and the usefulness score into the measurement index calculation module, calculate and output the measurement index of the paper to be tested, and the measurement index is used to measure the innovation quality of the paper to be tested.

2. The method for generating measurement indicators of paper innovation quality based on a large language model according to claim 1 is characterized in that: The paper data to be tested includes a text dataset of the paper title and abstract.

3. The method for generating measurement indicators of paper innovation quality based on a large language model according to claim 1 is characterized in that: The novelty measurement module includes an automatic keyword extraction submodule and a novelty score calculation submodule that are interconnected. The automatic keyword extraction submodule is used to construct a large language model, and the large language model is used to extract keywords from the paper data to be tested; the novelty score calculation submodule is used to calculate the novelty score of the paper to be tested based on the keywords.

4. The method for generating measurement indicators of paper innovation quality based on a large language model according to claim 3 is characterized in that: The large language model is a word vector generation model and a keyword recognition model that are connected to each other. The keyword extraction of the test paper data by using the large language model includes: S301. Input the test paper data into the word vector generation model for word segmentation to obtain multiple word segments, map each word segment into a high-dimensional vector space and convert it into a word vector to obtain a word vector set; S302. Taking the word vector set as an input sequence, using the keyword recognition model to capture complex patterns and long-distance dependencies in the input sequence, and obtaining a keyword set; S303. Take the keyword set as a high-dimensional vector, convert the high-dimensional vector into a dimensional vector required for the target task, use an activation function to convert the dimensional vector into a probability distribution of keywords, and divide the keywords into problem words and method words according to the probability distribution.

5. The method for generating measurement indicators of paper innovation quality based on a large language model according to claim 4 is characterized in that: The novelty score of the paper to be tested is calculated based on the keywords, including: S311. Take the publication year of the paper to be tested as the reference point, compare the problem words and method words in other papers published earlier than the paper to be tested in the same field, calculate the frequency of problem words and method words, and merge the frequency of problem words and method words into the frequency of keyword appearance. ; S312. Based on the frequency of keyword occurrence , the expression for calculating the novelty score is as follows: in, represents the novelty score of the paper to be tested, represents the keyword set of the paper to be tested, Representing a collection The number of elements of It represents the kth word that appears in the paper before it is published.

6. The method for generating measurement indicators of paper innovation quality based on a large language model according to any one of claims 1 to 5, characterized in that: The usefulness score of the paper to be tested is calculated based on the paper data to be tested, including: S321. Divide the citation frequency of the paper data to be tested into intervals and perform score processing to obtain the citation frequency score CPi; S322. Integrate and count the journals of the citing documents of the papers to be tested, and calculate the authoritative citation ratio of each citing document , and assign the authoritative citation ratio within the corresponding sub-area Citing article quality score ; S323. Combine the CPI score with the quality score of the citing literature Multiply them together to get the usefulness score of the paper to be tested. .

7. The method for generating measurement indicators of paper innovation quality based on a large language model according to claim 6 is characterized in that: The measurement indicators of the papers to be tested are calculated and output using the approximate ideal solution sorting method, including: S401. Based on the novelty score and the usefulness score, construct an initial judgment matrix ; S402. The initial judgment matrix Perform standardization processing to obtain a standardized matrix Z; S403. Determine the positive ideal solution of the novelty score and usefulness score in the normalized matrix Z and negative ideal solution ; S404. Based on the positive ideal solution and negative ideal solution , and the weighted Euclidean distance is calculated to obtain the distance from each measurement object to the positive ideal solution The positive ideal solution distance and negative ideal solution distance ; S405. Based on the positive ideal solution distance and negative ideal solution distance , calculate the distance to the positive ideal solution and negative ideal solution distance Relative proximity score ; S406. Score the relative proximity Define as the measurement indicators, arrange the measurement indicators, and complete the measurement of the innovation quality of the paper to be measured.

8. The method for generating measurement indicators of paper innovation quality based on a large language model according to claim 7 is characterized in that: The initial judgment matrix The form is as follows: in, Represents the initial judgment matrix The i Line m Column elements, i represents the number of measurement objects, m= 2 represents the number of measurement indicators, represents the novelty score of the paper to be tested, Represents the usefulness score of the paper to be tested.

9. The method for generating measurement indicators of paper innovation quality based on a large language model according to claim 8 is characterized in that: The initial judgment matrix After standardization, the calculation expression of the standardized matrix Z is as follows: After standardization, the standardized matrix Z is as follows: in, represents the first i Line m Column elements.

10. A measurement index generation system for the innovation quality of papers based on a large language model, characterized in that: include: The acquisition module is used to obtain the data of the paper to be tested; An input module, used for inputting the paper data to be tested into a preset scientific paper innovation quality measurement model, wherein the scientific paper innovation quality measurement model includes a novelty measurement module, a usefulness score calculation module and a measurement index calculation module, and the novelty measurement module and the usefulness score calculation module are connected to the measurement index calculation module in parallel; The measurement module is used to construct a large language model using the novelty measurement module, extract keywords from the paper data to be tested through the large language model, and calculate the novelty score of the paper to be tested based on the keywords; the usefulness score calculation module calculates the usefulness score of the paper to be tested based on the paper data to be tested; the novelty score and the usefulness score are input into the measurement index calculation module, and the measurement index calculation module calculates and outputs the measurement index of the paper to be tested, and the measurement index is used to measure the innovation quality of the paper to be tested.

Images