Literature quality evaluation method and device, equipment, medium and product

Abstract

The invention discloses a literature quality evaluation method and device, equipment, a medium and a product, and belongs to the technical field of literature quality evaluation.The method comprises the steps that multiple literatures in a target domain are obtained to construct a domain literature library of the target domain, and each literature in the domain literature library comprises a scientific quality score; performing uniform sampling from a domain literature library according to the scientific quality score to obtain a first literature data set; constructing a training sample set according to the first literature data set; the training sample set is adopted to supervise and finely adjust the large language model, the output of the large language model is a short reasoning chain, and the short reasoning chain is used for indicating the predicted quality scores of the two literatures and the relative quality relation of the two literatures; carrying out reinforcement learning on the fine-tuned large language model by adopting a GRPO algorithm; and performing literature quality evaluation according to the trained large language model. According to the method, efficient, accurate and reliable literature quality evaluation can be realized.

Description

Technical Field

[0001] This invention relates to the field of literature quality assessment technology, and in particular to a method, apparatus, equipment, medium and product for literature quality assessment. Background Technology

[0002] Literature quality assessment is a core component in evaluating the originality, importance, and rigor of research findings. It serves as a crucial foundation for job evaluation, promotion, research funding allocation, and project assessment, and is a significant driver of sustainable scientific research development. With the exponential growth in the number of publications, traditional manual assessment methods, due to their low efficiency, are no longer sufficient to meet the demands of evaluating the quality of a large volume of literature.

[0003] In related technologies, literature quality assessment methods are usually implemented through statistical analysis, such as assessing the quality of literature through indirect indicators (such as citation counts and journal impact factors).

[0004] However, literature quality assessment methods based on statistical analysis only analyze indirect indicators of literature. Indirect indicators are greatly affected by technical fields and publication time, and cannot truly reflect the overall quality of literature. Therefore, literature quality assessment methods that only consider indirect indicators have low accuracy. Summary of the Invention

[0005] This invention provides a method, apparatus, device, medium, and product for assessing the quality of literature, enabling efficient, accurate, and reliable assessment of literature quality. The technical solution includes at least the following components: Firstly, a method for assessing the quality of literature is provided, comprising: acquiring multiple documents within a target domain to construct a domain literature repository, wherein each document in the domain literature repository includes a scientific quality score, which is calculated based on impact factor, JCR partition, and cumulative citations; uniformly sampling from the domain literature repository based on the scientific quality scores to obtain a first literature dataset; constructing a training sample set based on the first literature dataset; supervising fine-tuning a large language model using the training sample set, wherein the output of the large language model is a short inference chain, which is used to indicate the predicted quality scores of two documents and the relative quality relationship between the two documents; performing reinforcement learning on the fine-tuned large language model using the GRPO algorithm, wherein the reward function during the reinforcement learning process includes an alignment reward and a relationship quality prediction reward; and assessing the quality of literature based on the trained large language model.

[0006] Optionally, the step of uniformly sampling from the domain literature database based on the scientific quality score to obtain the first literature dataset includes: uniformly dividing the scientific quality score of each document in the domain literature database into multiple scientific quality score intervals; and uniformly sampling from each scientific quality score interval to obtain the first literature dataset.

[0007] Optionally, constructing a training sample set based on the first literature dataset includes: constructing multiple past comparison pairs and multiple future comparison pairs for each focus document based on the first literature dataset, thereby obtaining multiple data pairs. The past comparison pairs include the focus document and a document cited by the focus document, and the future comparison pairs include the focus document and a document that cites the focus document. The focus document is a paper in the first literature dataset. Each data pair is labeled according to the independent citation probability, co-citation probability, and scientific quality score to obtain the training sample set.

[0008] Optionally, the first data pair is one of the plurality of data pairs, and the first data pair includes document A and document B. The step of labeling each data pair according to independent citation probability, co-citation probability, and scientific quality score includes: if the independent citation probability of document A is greater than the independent citation probability of document B, and the number of intervals in which the scientific quality score interval to which document A belongs exceeds the number of intervals in which the scientific quality score interval to which document B belongs is greater than X, then label the first data pair as having a higher quality for document A than for document B; if the co-citation probability of document A exceeds the co-citation probability of document B, and the number of intervals in which the scientific quality score interval to which document A belongs exceeds the number of intervals in which the scientific quality score interval to which document B belongs is less than X-1, then label the first data pair as having a quality approximately equal to the quality for document B; if the independent citation probability of document B is greater than the independent citation probability of document A, and the number of intervals in which the scientific quality score interval to which document B belongs exceeds the number of intervals in which the scientific quality score interval to which document A belongs is greater than X, then label the first data pair as having a higher quality for document B than for document A.

[0009] Optionally, in the reward function, the alignment reward is expressed by the following formula:

[0010] in, It is the prediction quality score output by the large language model for document A. This is the predicted quality score output by the large language model for document B, where document A and document B are a data pair input to the large language model. This is the scientific quality score of document A. This is the scientific quality score of document B. It is the alignment reward of document A. It is the alignment reward for document B; In the reward function, the relationship quality prediction reward is expressed by the following formula:

[0011] in, The reward is given for predicting the quality of the relationship. , representing the difference in the prediction quality scores of the large language model for document A and document B. This represents the difference in scientific quality scores between document A and document B. ,express and The absolute error, For piecewise reward functions, It is expressed by the following formula:

[0012] in, This represents the relative quality relationship between document A and document B predicted by the large language model. This indicates the relative quality relationship between cited document A and document B. , This indicates that the quality of document A is greater than the quality of document B. This indicates that the quality of document A is less than the quality of document B. This indicates that the quality of document A is approximately equal to the quality of document B.

[0013] Optionally, the step of evaluating the quality of documents based on the trained large language model includes: retrieving multiple comparative documents from the domain literature database based on the documents to be evaluated in the target domain; pairing the documents to be evaluated with each comparative document to obtain multiple data pairs to be evaluated; inputting the multiple data pairs to be evaluated into the trained large language model to obtain a short inference chain for each data pair to be evaluated; and performing weighted aggregation on the short inference chains of each data pair to obtain a weighted aggregated quality score for the documents to be evaluated.

[0014] Secondly, a document quality assessment device is also provided, comprising: an acquisition module for acquiring multiple documents within a target domain to construct a domain document library for the target domain, wherein each document in the domain document library includes a scientific quality score, which is calculated based on impact factor, JCR partition, and cumulative citations; a sampling module for uniformly sampling from the domain document library according to the scientific quality scores to obtain a first document dataset; a training set construction module for constructing a training sample set based on the first document dataset; a fine-tuning module for supervised fine-tuning of a large language model using the training sample set, wherein the output of the large language model is a short inference chain, which indicates the predicted quality scores of two documents and the relative quality relationship between the two documents; a reinforcement learning module for performing reinforcement learning on the fine-tuned large language model using the GRPO algorithm, wherein the reward function during the reinforcement learning process includes an alignment reward and a relation quality prediction reward; and a document quality assessment module for assessing the document quality based on the trained large language model.

[0015] Optionally, the sampling module is further configured to uniformly divide the scientific quality scores of each document in the domain literature database into multiple scientific quality score intervals; and uniformly sample from each scientific quality score interval to obtain the first literature dataset.

[0016] Optionally, the training set construction module is further configured to construct multiple past comparison pairs and multiple future comparison pairs for each focus document based on the first document dataset, thereby obtaining multiple data pairs. The past comparison pairs include the focus document and a document cited by the focus document, and the future comparison pairs include the focus document and a document that cites the focus document. The focus document is a paper in the first document dataset. Each data pair is labeled according to the independent citation probability, co-citation probability, and scientific quality score to obtain the training sample set.

[0017] Optionally, the first data pair is one of the plurality of data pairs, and the first data pair includes document A and document B. The training set construction module is further configured to: label the first data pair as having a higher quality for document A than for document B when the independent citation probability of document A is greater than that of document B, and the number of intervals in which the scientific quality score interval to which document A belongs exceeds the number of intervals in which the scientific quality score interval to which document B belongs is greater than X; label the first data pair as having a quality approximately equal to that of document B when the co-citation probability of document A exceeds that of document B, and the number of intervals in which the scientific quality score interval to which document A belongs exceeds the number of intervals in which the scientific quality score interval to which document B belongs is less than X-1; and label the first data pair as having a higher quality for document B than that of document A when the independent citation probability of document B is greater than that of document A, and the number of intervals in which the scientific quality score interval to which document B belongs exceeds the number of intervals in which the scientific quality score interval to which document A belongs is greater than X.

[0018] Optionally, in the reinforcement learning module, the alignment reward in the reward function is represented by the following formula:

[0019] in, It is the prediction quality score output by the large language model for document A. This is the predicted quality score output by the large language model for document B, where document A and document B are a data pair input to the large language model. This is the scientific quality score of document A. This is the scientific quality score of document B. It is the alignment reward of document A. It is the alignment reward for document B; In the reward function, the relationship quality prediction reward is expressed by the following formula:

[0020] in, The reward is given for predicting the quality of the relationship. , representing the difference in the prediction quality scores of the large language model for document A and document B. This represents the difference in scientific quality scores between document A and document B. ,express and The absolute error, For piecewise reward functions, It is expressed by the following formula:

[0021] in, This represents the relative quality relationship between document A and document B predicted by the large language model. This indicates the relative quality relationship between cited document A and document B. , This indicates that the quality of document A is greater than the quality of document B. This indicates that the quality of document A is less than the quality of document B. This indicates that the quality of document A is approximately equal to the quality of document B.

[0022] Optionally, the document quality assessment module is further configured to retrieve multiple comparative documents from the domain literature database based on the documents to be assessed in the target domain; pair the documents to be assessed with each comparative document to obtain multiple data pairs to be assessed; input the multiple data pairs to be assessed into the trained large language model to obtain a short inference chain for each data pair to be assessed; and perform weighted aggregation on the short inference chains of each data pair to be assessed to obtain a weighted aggregated quality score for the document to be assessed.

[0023] Thirdly, a computer device is also provided, comprising: a memory and a processor, wherein the memory stores at least one computer program, the at least one computer program being loaded and executed by the processor to perform the document quality assessment method described in the above embodiments.

[0024] Fourthly, a computer-readable storage medium is also provided, wherein at least one computer program is stored in the computer-readable storage medium, the at least one computer program being loaded and executed by a processor to perform the document quality assessment method described in the above embodiments.

[0025] Fifthly, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the method described in the first aspect.

[0026] The beneficial effects of the technical solution provided by this invention include at least the following: In this embodiment, by using a training sample set to supervise the fine-tuning of the large language model, the large language model acquires basic pairwise comparison capabilities and can generate short inference chains between two documents. By employing the GRPO algorithm to perform reinforcement learning on the fine-tuned large language model and designing a proprietary reward function based on document evaluation, including alignment rewards and relation quality prediction rewards, the scoring accuracy and scientific impact alignment of the short inference chains output by the large language model are effectively improved. This, in turn, enhances the accuracy of document quality evaluation, achieving efficient, accurate, and reliable document quality evaluation. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in this embodiment, the accompanying drawings used in the description of the embodiment will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 A flowchart of a document quality assessment method provided by an exemplary embodiment of the present invention is shown; Figure 2 It is the predicted quality score output by the trained large language model for documents with different JCR scores; Figure 3 A flowchart of a document quality assessment method provided by another exemplary embodiment of the present invention is shown; Figure 4 A schematic diagram of the structure of a document quality assessment device provided in an exemplary embodiment of the present invention is shown; Figure 5 This is a schematic diagram of the structure of a computer device provided in an exemplary embodiment of the present invention. Detailed Implementation

[0029] Unless otherwise defined, the technical or scientific terms used herein shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains. The terms “first,” “second,” “third,” and similar terms used in this patent application specification and claims do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms “an” or “a” and similar terms do not indicate a quantity limitation, but rather indicate the presence of at least one. The terms “comprising” or “including” and similar terms mean that the elements or objects preceding “comprising” or “including” encompass the elements or objects listed following “comprising” or “including” and their equivalents, but do not exclude other elements or objects.

[0030] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0031] Figure 1 A flowchart illustrating a document quality assessment method provided by an exemplary embodiment of the present invention is shown, which can be executed by a computer device. See also Figure 1 The method includes: In step 101, multiple documents within the target domain are acquired to construct a domain literature library for the target domain.

[0032] Each document in the domain literature database includes a science quality score, which is calculated based on impact factor, JCR quartile, and cumulative citations.

[0033] Here, the target field can be any field that requires literature quality assessment, and this embodiment does not limit the type of target field.

[0034] When building a domain literature database, for any given document, the following information needs to be collected: title, abstract, full-text PDF format, and multiple metrics.

[0035] Several indicators are included: cumulative citation count, journal impact factor, JCR (Journal Citation Reports) quartile, probability of independent citations, probability of co-citations, and scientific quality score. Of these indicators, only the scientific quality score is calculated.

[0036] Optionally, the science quality score is calculated using formula (1).

[0037] (1) In formula (1), This represents the scientific quality score of the document. For the JCR partition of the literature, Image factors for the literature. This represents the cumulative number of citations to the document.

[0038] It should be noted that in the above formula (1) , , These are indicators from the same document, based on the document's... , , The scientific quality score of the document can be calculated.

[0039] In step 102, a first literature dataset is obtained by uniformly sampling from the domain literature database based on the scientific quality score.

[0040] Optionally, step 102 includes steps 1021 to 1022 as follows.

[0041] Step 1021: Divide the scientific quality scores of each document in the domain literature database into multiple scientific quality score intervals evenly. In actual literature quality evaluation, literature is usually divided into different quality levels. However, the scientific quality score calculated by the above formula (1) is not an integer. Therefore, it is necessary to map the scientific quality score to different quality levels in order to normalize the scientific quality score.

[0042] For example, there are k quality levels, where k is a positive integer. All scientific quality scores can then be equally divided into k scientific quality score intervals in ascending order, with each interval having the same length. For example, k can be a positive integer such as 10 or 100. The value of k can be set according to requirements; this embodiment does not limit the value of k.

[0043] Except for the scientific quality score in step 1021, which is calculated using formula (1), the scientific quality scores involved in subsequent steps are all in the form of normalized quality grades. For example, there are a total of 1, 2, ..., k scientific quality score intervals. If the scientific quality score of a certain document calculated using formula (1) belongs to the second scientific quality score interval, then the scientific quality score of the document is finally "2".

[0044] Step 1022: Uniformly sample from each scientific quality score interval to obtain the first literature dataset.

[0045] Here, a first literature dataset can be obtained by uniformly sampling from each scientific quality score interval at a set ratio.

[0046] In step 103, a training sample set is constructed based on the first literature dataset.

[0047] Optionally, step 103 includes steps 1031 to 1032.

[0048] Step 1031: Based on the first literature dataset, construct multiple past comparison pairs and multiple future comparison pairs for each focus literature to obtain multiple data pairs.

[0049] Past comparisons included the focus paper and a paper cited by the focus paper. Future comparisons will include the focus paper and a paper that cites the focus paper. The focus paper is a paper in the first literature dataset.

[0050] In a first possible implementation, multiple papers can be manually selected from the first literature dataset as focus literature according to the requirements, and then multiple past comparison pairs and multiple future comparison pairs can be constructed based on the focus literature.

[0051] In a second possible implementation, multiple documents in the first document dataset can be sorted from largest to smallest according to the size of a certain index. Then, N documents are uniformly sampled from the sorted dataset as focus documents. Based on the focus documents, multiple past comparison pairs and multiple future comparison pairs are constructed.

[0052] In a third possible implementation, each paper in the first literature dataset can be used as a focus literature, and then multiple past comparison pairs and multiple future comparison pairs can be constructed based on the focus literature.

[0053] Past comparison pairs are constructed around the literature cited by the focus literature. A past comparison pair includes a comparison sample and a focus literature, where the comparison sample was published earlier than the focus literature and is cited by the focus literature. For example, a past comparison pair could be a paper A from 2010 and a paper A cited between 2000 and 2010. Past comparison pairs enable large language models to learn early in training whether a paper is more valuable than earlier research, and by how much.

[0054] Future comparison pairs are constructed around subsequent papers that cite the focus paper. A future comparison pair includes a comparison sample and a focus paper, where the comparison sample was published later than the focus paper and cites the focus paper. For example, using paper A from 2010 and a paper that cites paper A from 2010 to 2020 as a future comparison pair allows the large language model to further understand "how the value of old papers changes as new research emerges" during training.

[0055] The combined effect of past and future comparison pairs provides the large language model with a complete learning sample that "views value from the past and changes from the future".

[0056] Figure 2 It is the predicted quality score output by the trained large language model for documents with different JCR scores. Figure 2 The horizontal axis represents the publication year of the focus article (0). Negative numbers to the left of 0 represent the publication year of the comparison samples in the past comparison pair (e.g., -1 means the article was published one year before the focus article). Positive numbers to the right of 0 represent the publication year of the comparison samples in the future comparison pair (e.g., 1 means the article was published one year after the focus article). The vertical axis represents the prediction quality score output by the large language model.

[0057] like Figure 2As shown, the predicted quality scores output by the large language model for all JCR scores (JCR scores include Q1-Q4, corresponding to journals of different impact levels) decrease over time. This is because with past datasets, the model can recognize that "early-stage focus papers are more valuable than older papers," so it assigns higher scores on the left side of the time axis (e.g., 10 years before the focus paper's publication); while with future datasets, the model knows that "the value of older papers will gradually decrease compared to newer papers," so it assigns lower scores on the right side of the time axis (e.g., 10 years after the focus paper's publication). Moreover, the downward trend is almost consistent for both top-tier journals (Q1) and ordinary journals (Q4), indicating that through training with past and future comparison pairs, the model can not only distinguish the static differences between papers of different quality (Q1 scores are always higher than Q4), but also accurately capture the dynamic law that all papers depreciate due to knowledge updates. Ultimately, the model achieves the goal of "considering knowledge depreciation when evaluating paper value," verifying the effectiveness of the training results.

[0058] Step 1032: Label each data pair according to the independent citation probability, co-citation probability, and scientific quality score to obtain the training sample set.

[0059] For example, the first data pair is one of the plurality of data pairs, and the first data pair includes document A and document B. Step 1032 will be described below using the first data pair as an example.

[0060] Optionally, step 1032 includes the following three cases.

[0061] (1) If the independent citation probability of document A is greater than that of document B, and the number of intervals in which the scientific quality score interval to which document A belongs exceeds the number of intervals in which the scientific quality score interval to which document B belongs is greater than X, then the first data pair is labeled as having a higher quality for document A than for document B.

[0062] (2) If the co-citation probability of document A exceeds that of document B, and the number of intervals in which the scientific quality score interval of document A exceeds that of document B is less than X-1, then the first data pair is labeled as having a quality of document A that is approximately equal to that of document B.

[0063] (3) If the independent citation probability of document B is greater than that of document A, and the number of intervals in which the scientific quality score interval to which document B belongs exceeds that of the scientific quality score interval to which document A belongs is greater than X, then the first data pair is labeled as having a higher quality for document B than for document A.

[0064] X can be set based on experience, and X is a positive integer. For example, the value of X can range from 2 to 5, such as X being 2, 3, or 5.

[0065] For example, let X be 3. If the independent citation probability of document A is greater than that of document B, document A belongs to the 5th scientific quality score interval, and document B belongs to the 1st scientific quality score interval, then because 5-1 is greater than 3, the first data pair is labeled as having a higher quality for document A than for document B.

[0066] Conversely, X is set to 3. When the independent citation probability of document B is greater than that of document A, the scientific quality score interval to which document A belongs is the first scientific quality score interval, and the scientific quality score interval to which document B belongs is the fifth scientific quality score interval, then since 5-1 is greater than 3, the first data pair is labeled as having a higher quality for document B than for document A.

[0067] Similarly, if X is 3, then X-1=2. When the co-citation probability of document A exceeds that of document B, and document A belongs to the 5th scientific quality score interval while document B belongs to the 4th scientific quality score interval, then since 5-4 is less than 2, the first data pair is labeled as having a quality approximately equal to that of document B.

[0068] The relative quality relationship between two documents in all data pairs can be annotated using the above method. The annotation result can be simply expressed as follows: (Corresponding to case (1)) (Corresponding to case (2)) (Corresponding case (3)).

[0069] Here, the relative mass relationship includes ,in This indicates that the quality of document A is greater than the quality of document B. This indicates that the quality of document A is less than the quality of document B. This indicates that the quality of document A is approximately equal to the quality of document B.

[0070] In summary, each data pair in the training sample set includes: a focus article (containing the title, abstract, and full text), a comparison sample, the scientific quality score of the focus article, the scientific quality score of the comparison sample, and the relative quality relationship between the focus article and the comparison sample. The scientific quality scores of the focus article, the comparison sample, and the relative quality relationship between the focus article and the comparison sample are stored as labels; no labels are input during training.

[0071] If in a data pair, the publication year of the comparative sample is earlier than the publication year of the focus article, then the data pair is a past comparison pair; if in a data pair, the publication year of the comparative sample is later than the publication year of the focus article, then the data pair is a future comparison pair.

[0072] In addition, the scientific quality score of each data pair in the training sample set is an integer ranging from 1 to k, where k is a preset upper limit for scientific quality scores.

[0073] In step 104, the large language model is fine-tuned under supervision using the training sample set.

[0074] Before implementing step 104, the training sample set can be further filtered to improve its functionality. For example, a first prompt word and the titles and abstracts of two documents in a given data pair can be input into a large-scale language model. The first prompt word guides the large language model to evaluate the acceptability of the annotation results (i.e., the relative quality relationship in the data pair). The large language model will then output a binary judgment of "YES" or "NO". Data pairs with a result of "YES" can be retained, while data pairs with a result of "NO" can be discarded, thus filtering the training sample set.

[0075] In this embodiment, supervised fine-tuning (SFT) can be achieved by using efficient parameter fine-tuning methods such as QloRA (Quantized Low-Rank Adaptation) and LoRA (Low-Rank Adaptation).

[0076] During supervised fine-tuning, the original parameters of the underlying LLM stored in the quantized memory are frozen, and only a small number of new parameters introduced by the parameter-efficient fine-tuning algorithm are updated to improve training efficiency and mitigate the catastrophic forgetting problem.

[0077] During training, the input to the large language model is the second cue word, the titles and abstracts of the two papers in the data pair; the output of the large language model is a short inference chain, which is used to indicate the predicted quality scores of the two documents in the data pair and the relative quality relationship between the two documents.

[0078] The second cue word is used to guide the large language model to score the two documents in the data pair to obtain a predicted quality score, and to guide the large language model to evaluate the relative quality relationship between the two documents in the data pair.

[0079] When generating short inference chains, the large language model first generates the predicted quality score of the focus paper, then generates the relative quality relationship between the focus paper and the comparison samples, and finally generates the predicted quality score of the comparison samples. In the process of generating short inference chains, the predicted quality score of the focus paper and the relative quality relationship between the focus paper and the comparison samples generated earlier will form a constraint relationship on the predicted quality score of the comparison samples generated later.

[0080] In the short inference chain, the label corresponding to the predicted quality score of the focus paper is its scientific quality score, and the label corresponding to the predicted quality score of the comparison sample is its scientific quality score. The label for the relative quality relationship between the focus paper and the comparison sample is the relative quality relationship indicated in step 1032 above.

[0081] It should be noted that the prediction quality score in the short inference chain output by the large language model is an integer ranging from 1 to k.

[0082] In step 105, the GRPO algorithm is used to perform reinforcement learning on the fine-tuned large language model.

[0083] Supervised fine-tuning enables large language models to acquire basic evaluation capabilities and memorize training samples, but their generalization ability is insufficient. Therefore, in step 105, the GRPO (Group Relative Policy Optimization) algorithm of reinforcement learning is used to further enhance the generalization ability of large language models and improve their comparative analysis capabilities.

[0084] There are many related technologies regarding the implementation of reinforcement learning using the GRPO algorithm, so detailed descriptions are omitted here. This embodiment only describes the reward function used in the reinforcement learning process using the GRPO algorithm.

[0085] In this embodiment, the reward function in the reinforcement learning process includes: alignment reward, relation quality prediction reward, and standardized output reward. The reward function is used to provide feedback during training and helps the model understand the numerical relationships between scores. These rewards are explained below.

[0086] The alignment reward is expressed by the following formula (2).

[0087] (2) In formula (2), It is the prediction quality score of the large language model for document A. This is the predicted quality score of the large language model for document B. Documents A and B are a data pair input to the large language model. This is the scientific quality score of document A. This is the scientific quality score of document B. This is the alignment reward for document A, used to measure the degree of alignment between the predicted quality score of the large language model's output for document A and the actual score (i.e., the scientific quality score of document A). Reward "accurate predictions" and punish "prediction errors". This is the alignment reward for document B, used to measure the degree of alignment between the predicted quality score of the large language model's output for document B and the actual score (i.e., the scientific quality score of document B). Reward "accurate predictions" and punish "prediction errors".

[0088] In the reward function, the reward for predicting relation quality is expressed by the following formula (3).

[0089] (3) In formula (3), Rewards for predicting relationship quality. , representing the difference in prediction quality scores between the large language model for document A and document B. This represents the difference in scientific quality scores between document A and document B. ,express and The absolute error, This is a piecewise reward function. When k is 10, It is represented by the following formula (4).

[0090] (4) In formula (4), This represents the relative quality relationship between document A and document B predicted by the large language model. This indicates the relative quality relationship between cited document A and document B. , This indicates that the quality of document A is greater than the quality of document B. This indicates that the quality of document A is less than the quality of document B. This indicates that the quality of document A is approximately equal to the quality of document B.

[0091] Is the piecewise function's prediction based on the relative quality relationship correct? or ) and the relative quality relationship between document A and document B in the labels ( Different levels of rewards are applied. This can be viewed as a "reward adjustment switch." The relative quality relationship of the output of the large language model is correct ( In cases where the error is small, the switch is loose (high reward); conversely, a large error results in a tight switch (low reward). This is because the relative quality relationship in the output of the large language model is incorrect. In cases where the error is small, the switch will be "half-tight" (the reward is not high). This piecewise reward function not only requires the model to accurately predict the scientific quality score of a single document, but also to capture the relative difference (score difference) between the scientific quality scores of document A and document B, while constraining the "reward cap when the relationship prediction is wrong" to prevent the model from obtaining high rewards through "extreme score differences" (i.e., "shortcut behavior").

[0092] The standardized output reward is expressed by formula (5).

[0093] (5) In formula (5), The standardized output reward is used to encourage the relative quality relationships of the large language model's output to conform to a preset standardized form. If the output of the large language model does not conform to the standardized form, a penalty is imposed. The standardized output reward forces the model to generate standardized output, ensuring that subsequent score extraction and relationship judgment can be automated (avoiding invalid training data due to format confusion).

[0094] The above reward functions can be aggregated to obtain the total reward function in GRPO reinforcement learning. This total reward function is expressed by formula (6).

[0095] (6) In formula (6), This is the total reward function. This total reward function also needs to be standardized using Z-Score to ensure dimensional consistency.

[0096] In step 106, the document quality is assessed based on the trained large language model.

[0097] Through steps 101 to 105 above, the large language model is trained in the field of document quality assessment. The trained large language model can then be used for document quality assessment in step 106.

[0098] In this embodiment, by using a training sample set to supervise the fine-tuning of the large language model, the large language model acquires basic pairwise comparison capabilities and can generate short inference chains between two documents. By employing the GRPO algorithm to perform reinforcement learning on the fine-tuned large language model and designing a proprietary reward function based on document evaluation, including alignment rewards and relation quality prediction rewards, the scoring accuracy and scientific impact alignment of the short inference chains output by the large language model are effectively improved. This, in turn, enhances the accuracy of document quality evaluation, achieving efficient, accurate, and reliable document quality evaluation.

[0099] Figure 3 A flowchart of a document quality assessment method provided by another exemplary embodiment of the present invention is shown, which can be executed by a computer device. See also Figure 3 The method includes: In step 301, multiple comparative documents are retrieved from the domain literature database based on the literature to be evaluated in the target domain.

[0100] In implementation, the document to be evaluated can be vectorized to extract its semantic vector. Then, based on this semantic vector, a search is performed in the domain literature database to obtain the top M comparable documents. Here, the documents in the domain literature database have already been vectorized in advance, so it is only necessary to calculate the semantic similarity between the semantic vector of the document to be evaluated and the semantic vectors of the documents in the domain literature database pairwise to obtain the top M comparable documents with the semantic vector of the document to be evaluated.

[0101] In addition, a time range can be set based on the publication date of the literature to be evaluated to improve the efficiency of searching for comparative literature. For example, the time range can be defined with the publication date of the literature to be evaluated as the center.

[0102] In step 302, the literature to be evaluated is paired with each comparative literature to obtain multiple data pairs to be evaluated.

[0103] Each data pair to be evaluated includes a literature to be evaluated and a comparative literature. Here, the data pair to be evaluated is equivalent to the data pair in steps 101 to 106 above, the literature to be evaluated is equivalent to the focus literature (i.e., literature A), and the comparative literature is equivalent to the comparative sample (i.e., literature B).

[0104] In step 303, multiple data pairs to be evaluated are input into the trained large language model to obtain a short inference chain for each data pair to be evaluated.

[0105] The trained large language model is obtained using the methods described in steps 101 to 106.

[0106] After the large language model is trained, the short inference chains it outputs are highly reliable. Simply input the title and summary of each data pair to be evaluated into the large language model to obtain the short inference chains output by the model for each pair.

[0107] In step 304, the short inference chains of each data pair to be evaluated are weighted and aggregated to obtain the weighted aggregated quality score of the document to be evaluated.

[0108] When performing weighted aggregation, it is necessary to calculate the absolute aggregation score and the relative aggregation score.

[0109] The absolute aggregate score of all data pairs to be evaluated is calculated using formula (7). In this embodiment, k is 10, which means that the prediction quality score of any document in the short inference chain output by the large language model is an integer between 1 and 10.

[0110] (7) In formula (7), For absolute aggregate fractions, The total number of data pairs to be evaluated. denoted as the prediction quality score of the document to be evaluated in the short inference chain of the i-th data pair to be evaluated. The predicted quality score of the literature to be evaluated in the short inference chain of the i-th data pair to be evaluated is linearly normalized from 1-10 to 0-2.

[0111] This normalization process aims to map the raw quality scores (range 1-10) of the documents to be evaluated to a standardized absolute quality interval [0, 2]. Here, 0 points represents the lowest scientific quality level, 2 points represents the highest scientific quality level, and 1 point represents the average scientific quality. This mapping eliminates the dimensional influence of the raw quantiles, allowing for the calculation of a standardized aggregate absolute score. This allows for a quantitative reflection of the scientific value of the literature.

[0112] The relative aggregation score of the predicted quality score of the literature to be evaluated in the short inference chain of the i-th data pair to be evaluated is expressed by formula (8).

[0113] (8) In formula (8), This represents the predicted quality score of the comparative literature in the short inference chain of the i-th data pair to be evaluated. The short inference chain of the i-th data pair to be evaluated includes... , , In three cases, formula (8) can be used to transform each short inference chain according to the different cases, so as to obtain the relative aggregation score of each short inference chain.

[0114] After obtaining the absolute aggregation score and the relative aggregation score, weighted aggregation is performed using formula (9) to obtain the weighted aggregation quality score of the literature to be evaluated.

[0115] (9) In formula (9), The weighted aggregation quality score of the literature to be evaluated. These are the preset weighting coefficients.

[0116] Thus, through weights It integrates "absolute aggregate score" and "relative aggregate score", balancing "the accuracy of the self-score of the document to be evaluated" and "the quality positioning of the document to be evaluated relative to other papers", so as to accurately obtain the final quality score of the document to be evaluated, namely the weighted aggregate quality score.

[0117] Traditional literature quality assessment methods judge the quality of a literature by the cosine similarity between the semantic vectors of the literature to be assessed and those of comparison literature. This approach only considers semantic differences and cannot accurately measure the innovative value of a paper. Furthermore, the encoding of semantic vectors is susceptible to anisotropy interference (the process of converting the text of a paper into low-dimensional vectors using an NLP model to quantify semantics; anisotropy refers to the uneven distribution of these vectors in space, with excessive clustering in a specific direction or region). This anisotropy interference causes the vectors of papers with large semantic differences to be close together, while the vectors of papers with similar semantics are far apart. This leads to distortion in the calculation of semantic similarity based on vectors, failing to truly reflect the semantic connections between papers. Consequently, the literature quality assessment method based on cosine similarity results in a bias in measuring the innovative value of a literature.

[0118] In this embodiment, through the above steps 101 to 106 and steps 301 to 304, objective and fair evaluation results are provided for literature evaluation, which helps to optimize the existing scientific research quality evaluation system and promote the automation and intelligent development of scientific evaluation.

[0119] The following are device embodiments of this application. For details not described in detail in the device embodiments, please refer to the above method embodiments.

[0120] Figure 4 A schematic diagram of a document quality assessment apparatus provided in an exemplary embodiment of the present invention is shown. See also Figure 4 The document quality assessment device 400 includes: an acquisition module 401, a sampling module 402, a training set construction module 403, a fine-tuning module 404, a reinforcement learning module 405, and a document quality assessment module 406.

[0121] The acquisition module 401 is used to acquire multiple documents in the target field to construct a domain literature library for the target field. Each document in the domain literature library includes a science quality score, which is calculated based on the impact factor, JCR partition, and cumulative citations.

[0122] The sampling module 402 is used to uniformly sample from the domain literature database according to the scientific quality score to obtain the first literature dataset.

[0123] The training set construction module 403 is used to construct a training sample set based on the first literature dataset.

[0124] The fine-tuning module 404 is used to perform supervised fine-tuning of the large language model using the training sample set. The output of the large language model is a short inference chain, which is used to indicate the predicted quality scores of two documents and the relative quality relationship between the two documents.

[0125] The reinforcement learning module 405 is used to perform reinforcement learning on the fine-tuned large language model using the GRPO algorithm. During the reinforcement learning process, the reward function includes alignment reward and relation quality prediction reward.

[0126] The document quality assessment module 406 is used to assess the quality of documents based on the trained large language model.

[0127] Optionally, the sampling module 402 is also used to uniformly divide the scientific quality scores of each document in the domain literature database into multiple scientific quality score intervals; and to uniformly sample from each scientific quality score interval to obtain the first literature dataset.

[0128] Optionally, the training set construction module 403 is further configured to construct multiple past comparison pairs and multiple future comparison pairs for each focus document based on the first document dataset, thereby obtaining multiple data pairs. The past comparison pairs include the focus document and a document cited by the focus document, and the future comparison pairs include the focus document and a document that cites the focus document. The focus document is a paper in the first document dataset. Each data pair is labeled according to the independent citation probability, co-citation probability, and scientific quality score to obtain a training sample set.

[0129] Optionally, the first data pair is one of multiple data pairs, including document A and document B. The training set construction module 403 is further configured to: label the first data pair as having a higher quality for document A than for document B if the independent citation probability of document A is greater than that of document B, and the number of intervals in which the scientific quality score interval to which document A belongs exceeds the number of intervals in which the scientific quality score interval to which document B belongs is greater than X; label the first data pair as having a quality approximately equal to that of document B if the co-citation probability of document A is greater than that of document B, and the number of intervals in which the scientific quality score interval to which document A belongs exceeds the number of intervals in which the scientific quality score interval to which document B belongs is less than X-1; and label the first data pair as having a higher quality for document B than that of document A if the independent citation probability of document B is greater than that of document A, and the number of intervals in which the scientific quality score interval to which document B belongs exceeds the number of intervals in which the scientific quality score interval to which document A belongs is greater than X.

[0130] Optionally, in reinforcement learning module 405, the alignment reward in the reward function is represented by the following formula:

[0131] in, It is the prediction quality score of the large language model for document A. This is the predicted quality score of the large language model for document B. Documents A and B are a data pair input to the large language model. This is the scientific quality score of document A. This is the scientific quality score of document B. It is the alignment reward of document A. It is the alignment reward for document B; In the reward function, the reward for predicting relationship quality is expressed by the following formula:

[0132] in, Rewards for predicting relationship quality. , representing the difference in prediction quality scores between the large language model for document A and document B. This represents the difference in scientific quality scores between document A and document B. ,express and The absolute error, For piecewise reward functions, It is expressed by the following formula:

[0133] in, This represents the relative quality relationship between document A and document B as predicted by the large language model. This indicates the relative quality relationship between cited document A and document B. , This indicates that the quality of document A is greater than the quality of document B. This indicates that the quality of document A is less than the quality of document B. This indicates that the quality of document A is approximately equal to the quality of document B.

[0134] Optionally, the document quality assessment module 406 is also used to retrieve multiple comparative documents from the domain document database based on the documents to be assessed in the target domain; pair the documents to be assessed with each comparative document to obtain multiple data pairs to be assessed; input the multiple data pairs to be assessed into the trained large language model to obtain a short inference chain for each data pair to be assessed; and perform weighted aggregation on the short inference chains of each data pair to be assessed to obtain a weighted aggregated quality score for the documents to be assessed.

[0135] It should be noted that the document quality assessment device provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the document quality assessment device and the document quality assessment method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0136] The module division in this embodiment of the invention is illustrative and represents only one logical functional division. In actual implementation, other division methods are possible. Furthermore, the functional modules in each embodiment of the invention can be integrated into a single processor, exist as separate physical entities, or consist of two or more modules integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0137] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a terminal device (which may be a personal computer, mobile phone, or communication device, etc.) or processor to execute all or part of the steps of the method of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0138] Figure 5 This is a schematic diagram of the structure of a computer device provided in an exemplary embodiment of the present invention. For example... Figure 5 As shown, the computer device 500 includes a processor 501 and a memory 502.

[0139] Processor 501 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 501 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 501 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 501 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 501 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0140] The memory 502 may include one or more computer-readable storage media, which may be non-transitory. The memory 502 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 502 are used to store at least one instruction, which is executed by the processor 501 to implement the document quality assessment method provided in the embodiments of the present invention.

[0141] Those skilled in the art will understand that Figure 5 The structure shown does not constitute a limitation on the computer device 500, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0142] This invention also provides a non-transitory computer-readable storage medium, wherein when the instructions in the storage medium are executed by the processor of a computer device, the computer device is able to execute the document quality assessment method provided in this invention.

[0143] This invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the document quality assessment method provided in this invention.

[0144] The above description is merely an optional embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for assessing the quality of literature, characterized in that, The method includes: Multiple documents within a target field are acquired to construct a domain literature library for the target field. Each document in the domain literature library includes a science quality score, which is calculated based on impact factor, JCR partition, and cumulative citations. A first literature dataset is obtained by uniformly sampling from the literature database in the field based on the scientific quality score. Construct a training sample set based on the first literature dataset; The large language model is fine-tuned under supervision using a training sample set. The output of the large language model is a short inference chain, which is used to indicate the predicted quality scores of two documents and the relative quality relationship between the two documents. The GRPO algorithm is used to perform reinforcement learning on the fine-tuned large language model. In the reinforcement learning process, the reward function includes alignment reward and relation quality prediction reward. Literature quality assessment is performed based on the trained large language model.

2. The method according to claim 1, characterized in that, The process of uniformly sampling from the domain literature database based on the scientific quality score to obtain the first literature dataset includes: The scientific quality score of each document in the literature database in the field is evenly divided into multiple scientific quality score intervals; The first literature dataset is obtained by uniformly sampling from each scientific quality score interval.

3. The method according to claim 2, characterized in that, The step of constructing a training sample set based on the first document dataset includes: Based on the first literature dataset, multiple past comparison pairs and multiple future comparison pairs are constructed for each focus document to obtain multiple data pairs. The past comparison pair includes the focus document and a document cited by the focus document, and the future comparison pair includes the focus document and a document that cites the focus document. The focus document is a paper in the first literature dataset. The training sample set is obtained by labeling each data pair with its independent citation probability, co-citation probability, and scientific quality score.

4. The method according to claim 3, characterized in that, The first data pair is one of the plurality of data pairs, and the first data pair includes document A and document B. The annotation of each data pair based on independent citation probability, co-citation probability, and scientific quality score includes: If the independent citation probability of document A is greater than that of document B, and the number of intervals in which the scientific quality score interval to which document A belongs exceeds the number of intervals in which the scientific quality score interval to which document B belongs is greater than X, then the first data pair is labeled as having a higher quality for document A than for document B. If the co-citation probability of document A exceeds that of document B, and the number of intervals in which the scientific quality score interval to which document A belongs exceeds that of the scientific quality score interval to which document B belongs is less than X-1, then the first data pair is labeled as having a quality of document A that is approximately equal to that of document B. If the independent citation probability of document B is greater than that of document A, and the number of intervals in which the scientific quality score interval to which document B belongs exceeds the number of intervals in which the scientific quality score interval to which document A belongs is greater than X, then the first data pair is labeled as having a higher quality for document B than for document A.

5. The method according to any one of claims 1 to 3, characterized in that, In the reward function, the alignment reward is expressed by the following formula: in, It is the prediction quality score output by the large language model for document A. This is the predicted quality score output by the large language model for document B, where document A and document B are a data pair input to the large language model. This is the scientific quality score of document A. This is the scientific quality score of document B. It is the alignment reward of document A. It is the alignment reward for document B; In the reward function, the relationship quality prediction reward is expressed by the following formula: in, The reward is given for predicting the quality of the relationship. , representing the difference in the prediction quality scores of the large language model for document A and document B. This represents the difference in scientific quality scores between document A and document B. ,express and The absolute error, For piecewise reward functions, It is expressed by the following formula: in, This represents the relative quality relationship between document A and document B predicted by the large language model. This indicates the relative quality relationship between cited document A and document B. , This indicates that the quality of document A is greater than the quality of document B. This indicates that the quality of document A is less than the quality of document B. This indicates that the quality of document A is approximately equal to the quality of document B.

6. The method according to any one of claims 1 to 4, characterized in that, The document quality assessment based on the trained large language model includes: Based on the literature to be evaluated in the target field, multiple comparative literatures are retrieved from the literature database of the stated field; The document to be evaluated is paired with each comparative document to obtain multiple data pairs to be evaluated. The multiple data pairs to be evaluated are input into the trained large language model to obtain a short inference chain for each data pair to be evaluated. The short inference chains of each data pair to be evaluated are weighted and aggregated to obtain the weighted aggregated quality score of the document to be evaluated.

7. A document quality assessment device, characterized in that, The device includes: The acquisition module is used to acquire multiple documents in the target field to construct a domain literature library for the target field. Each document in the domain literature library includes a science quality score, which is calculated based on impact factor, JCR partition and cumulative citations. A sampling module is used to uniformly sample from the domain literature database according to the scientific quality score to obtain a first literature dataset; The training set construction module is used to construct a training sample set based on the first literature dataset. The fine-tuning module is used to supervise the fine-tuning of a large language model using a training sample set. The output of the large language model is a short inference chain, which is used to indicate the predicted quality scores of two documents and the relative quality relationship between the two documents. The reinforcement learning module is used to perform reinforcement learning on the fine-tuned large language model using the GRPO algorithm. In the reinforcement learning process, the reward function includes alignment reward and relation quality prediction reward. The document quality assessment module is used to assess the quality of documents based on the trained large language model.

8. A computer device, characterized in that, The computer device includes a memory and a processor, wherein the memory stores at least one computer program, which is loaded and executed by the processor to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement the method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the method described in any one of claims 1 to 6.

Images