A method and device for quantitatively evaluating a paper value

Abstract

The application discloses a kind of paper value quantitative evaluation method and device, it is related to data processing technical field, comprising: based on large language model, extract semantic element set from target paper, wherein, extraction step contains first-order prompt word extraction, and two-stage test screening, screening mode includes adversarial test and knowledge diffusion trajectory analysis;Domain benchmark library is constructed and maintained;Set the evaluation dimension of paper value quantification, execute multidimensional joint score based on the semantic element set and the domain benchmark library, determine paper value score.The present application solves the technical problems that the paper value evaluation of prior art relies on external citation data and cannot directly quantify the analysis of paper content, through semantic element extraction and multidimensional joint scoring mechanism, to achieve the technical effect of direct, objective and accurate quantitative evaluation of paper content value.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a method and apparatus for quantitatively evaluating the value of academic papers. Background Technology

[0002] The evaluation of a paper's value typically relies on external indicators such as citation count and journal ranking, which fail to accurately reflect the paper's actual academic value in terms of originality, research question formulation, and expression of viewpoints. Furthermore, these indicators suffer from data lag, limited coverage, and susceptibility to external interference, making it impossible to directly and objectively quantify the academic contribution of the paper's content itself. Summary of the Invention

[0003] This application provides a method and apparatus for quantitatively evaluating the value of academic papers, which addresses the technical problem that existing methods for evaluating the value of academic papers rely on external citation data and cannot directly quantify the content of the papers.

[0004] In view of the above problems, this application provides a method and apparatus for quantitatively evaluating the value of papers.

[0005] A first aspect of this application provides a method for quantitatively evaluating the value of a paper, the method comprising: Based on a large language model, a semantic element set is extracted from the target paper. The extraction steps include first-order prompt word extraction and second-order test screening, with screening methods including adversarial testing and knowledge diffusion trajectory analysis. A domain benchmark library is constructed and maintained. Evaluation dimensions for quantifying the paper's value are set, and a multi-dimensional joint scoring based on the semantic element set and the domain benchmark library is performed to determine the paper's value score. Specifically, dedicated scoring formulas are used for the dimensions of innovation, author's viewpoint, research question, and concept definition.

[0006] A second aspect of this application provides a device for quantifying the value of a paper, the device comprising: The semantic element extraction module is used to extract a set of semantic elements from the target paper based on a large language model. The extraction steps include first-order prompt word extraction and second-order test screening. The screening methods include adversarial testing and knowledge diffusion trajectory analysis. The benchmark library construction module is used to build and maintain a domain benchmark library. The scoring module is used to set the evaluation dimensions for quantifying the paper's value, and to perform multi-dimensional joint scoring based on the semantic element set and the domain benchmark library to determine the paper's value score. The scoring is calculated using dedicated scoring formulas for the dimensions of innovation, author's viewpoint, research question, and concept definition.

[0007] One or more technical solutions provided in this application have at least the following technical effects or advantages: This application, based on a large language model, extracts a semantic element set from target papers. The extraction process includes first-order prompt word extraction and second-order testing and screening, with screening methods including adversarial testing and knowledge diffusion trajectory analysis. It also constructs and maintains a domain benchmark library; sets evaluation dimensions for quantifying paper value; and performs multi-dimensional joint scoring based on the semantic element set and the domain benchmark library to determine the paper's value score. The scoring is performed using dedicated scoring formulas for the dimensions of innovation, author viewpoints, research questions, and concept definitions. This invention addresses the technical problem of existing paper value evaluation relying on external citation data and being unable to directly quantify the paper's content. Through semantic element extraction and a multi-dimensional joint scoring mechanism, it achieves the technical effect of directly, objectively, and accurately quantifying the value of paper content. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 A schematic diagram of a paper value quantification evaluation method provided in this application embodiment; Figure 2 This is a schematic diagram of a paper value quantification evaluation device provided in an embodiment of this application.

[0010] Figure labeling: Semantic element extraction module 11, benchmark library construction module 12, scoring module 13. Detailed Implementation

[0011] This application provides a method and apparatus for quantitatively evaluating the value of academic papers. It addresses the technical problem that existing methods for evaluating the value of academic papers rely on external citation data and cannot directly quantify the content of the papers. By using semantic element extraction and a multi-dimensional joint scoring mechanism, it achieves the technical effect of directly, objectively, and accurately quantifying the value of the paper content.

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0013] It should be noted that any variation of the terms "comprising" and "having" is intended to cover non-exclusive inclusion, for example, a process, method, apparatus, product, or server that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or modules that are not explicitly listed or that are inherent to such process, method, product, or apparatus.

[0014] Example 1, as Figure 1 As shown, this application provides a method for quantitatively evaluating the value of a paper, the method comprising: Step S100: Based on the large language model, extract the semantic element set from the target paper. The extraction steps include first-order prompt word extraction and second-order test screening. The screening methods include adversarial testing and knowledge diffusion trajectory analysis.

[0015] In this embodiment, first-order prompt word extraction is performed based on the first processing layer within the large language model. Firstly, the target paper's content is semantically parsed using a large language model based on the Transformer architecture. Through the synergistic effect of prompt word engineering, large language model fine-tuning, and preset extraction strategies, the semantic elements of the paper are extracted. Prompt word engineering is used to construct structured instruction templates during the model inference stage, guiding the large language model to understand the semantic levels of the paper's content through preset questions and example outputs. For example, prompts such as "Please extract the sentences in the text that demonstrate the innovative points" or "Find the paragraphs that describe the research problem" are designed to enable the model to automatically identify different types of semantic information when reading the paper.

[0016] Subsequently, fine-tuning of the large language model enhances its adaptability to academic text contexts. During this fine-tuning phase, parameters are adjusted based on labeled academic paper samples, enabling the model to more accurately understand the logical structure and expressive features within academic papers. For example, in the innovative sentence recognition task, the model enhances its ability to identify innovative content by learning typical semantic features such as "this study is the first of its kind." After fine-tuning, the large language model possesses the ability to distinguish between innovative sentences, opinion sentences, research question sentences, concept definition sentences, and unresolved problem sentences, enabling general extraction across papers from different fields.

[0017] Finally, a pre-defined extraction strategy is employed to semantically filter and optimize the output of the large language model. This strategy includes a semantic similarity-based filtering method and keyword triggering rules to eliminate redundant information, unify expression formats, and ensure semantic accuracy. For example, when the model extracts multiple similar research question sentences, duplicate content is merged through semantic similarity calculation; when different sentence structures express the same concept definition, the most representative sentence is retained through rule comparison. Through this extraction process, an initial set of semantic elements is generated, including innovative sentences, opinion sentences, research question sentences, concept definition sentences, unresolved questions, and paper domain classifications.

[0018] After obtaining the initial semantic element set, the second processing layer within the large language model optimizes the set, performing adversarial testing and knowledge diffusion trajectory prediction based on counterfactual trajectory perturbations. In this process, a small semantic perturbation is introduced through a counterfactual trajectory generator, and a counterfactual trajectory judge measures the conditional independence between latent variables and the robustness of elements across semantic, logical, and empirical dimensions to determine the first adversarial test data. Simultaneously, a knowledge predictor is used to predict knowledge diffusion trajectories, classifying them into three diffusion modes: exponential growth, logical growth, and impulsive decay, to determine the second diffusion test data. The initial semantic element set is then filtered by combining these two types of test data, eliminating invalid and redundant content, and weighting elements based on their robustness and propagation value to ultimately determine the final semantic element set.

[0019] Furthermore, the method provided in the application embodiments, in extracting a semantic element set from the target paper, further includes: Based on the first processing layer within the large language model, and using prompt word engineering, large language model fine-tuning, and preset extraction strategies, an initial set of semantic elements is extracted from the target paper. Based on the second processing layer within the large language model, adversarial testing based on counterfactual trajectory perturbation and knowledge diffusion trajectory prediction are performed to filter and weight the initial set of semantic elements, thereby determining the set of semantic elements.

[0020] In this embodiment, based on the first processing layer within the large language model, the information extraction direction is clarified by prompt word engineering, the large language model is fine-tuned to adapt to academic text scenarios, and the result quality is optimized by preset extraction strategies. The three work together to extract an initial set of semantic elements from the target paper.

[0021] Next, adversarial testing and knowledge diffusion trajectory prediction based on counterfactual trajectory perturbations are performed in the second processing layer within the large language model. In this process, adversarial testing introduces subtle semantic perturbations through a counterfactual trajectory generator, and uses a counterfactual trajectory judge to measure the conditional independence between latent variables and the robustness of elements including semantic, logical, and empirical dimensions, thus determining the first adversarial test data. Knowledge diffusion trajectory prediction is performed through a knowledge predictor, classifying diffusion into three modes: exponential growth, logical growth, and impulsive decay, to determine the second diffusion test data.

[0022] After completing adversarial testing and knowledge diffusion trajectory prediction, the initial semantic element set was screened and weighted. During the screening process, an initial screening was conducted based on the first adversarial test data, while simultaneously clarifying the rules for determining the scores for each dimension. The semantic dimension scores were calculated by converting the semantic similarity before and after perturbation. A similarity of 0.8 was set as the maximum score threshold; for every 0.1 decrease in similarity, the score decreased by 1 point, with a maximum score of 10 points. For example, a similarity of 0.9 yielded 10 points, 0.8 yielded 10 points, 0.7 yielded 9 points, and so on, until a similarity of 0.0 yielded 1 point, intuitively reflecting the stability of the elements against text perturbation. The logical dimension score is determined through an assessment of the completeness of the premise-conclusion chain. A logical closed-loop detection index system is constructed, covering three sub-indicators: premise rationality, reasoning coherence, and conclusion validity. Each sub-indicator has a maximum score of 3.33 points, for a total of 10 points. A perfect logical closed-loop requirement earns 10 points; minor logical flaws deduct 1-2 points; significant logical loopholes deduct 3-5 points; and severe logical breaks deduct 6-9 points. This quantifies the vulnerability of the premise-conclusion chain. The empirical dimension score is determined through a data reproducibility assessment, evaluating three dimensions: data source reliability, experimental procedure standardization, and variable control completeness. Each dimension has a maximum score of 3.33 points, for a total of 10 points. A perfect reproducibility standard in the domain earns 10 points; minor non-standard items deduct 1-2 points; non-compliance with key steps deducts 3-5 points; and unverifiable core data or procedures deduct 6-9 points. A robustness quantification threshold of at least 8 points is set for each dimension. Elements failing to reach the threshold in any dimension are removed, retaining only highly resistant core elements that meet the requirements in all three dimensions.

[0023] A second round of screening was conducted based on the second diffusion test data. Elements with pulse decay diffusion patterns were eliminated because they have no long-term academic impact after short-term propagation. Only elements with exponential growth and logical growth diffusion patterns that can provide effective support for the development of the field were retained, thus completing the dual screening process.

[0024] The subsequent weighting process employs a tiered weighting mechanism. First, a robustness score is calculated based on the scores of each dimension, with each of the three dimensions having a maximum score of 10 points, resulting in a total score of 30 points. Then, robustness levels are determined based on the total score, and basic weights are assigned. A total score of 27 points or higher qualifies as Level 1, indicating excellent performance across all three dimensions, with a basic weight of 0.4; 24 to 26 points qualifies as Level 2, indicating generally good performance, with a basic weight of 0.3; 21 to 23 points qualifies as Level 3, indicating that the corresponding element basically meets robustness requirements, with a basic weight of 0.2; and 20 points or lower qualifies as Level 4, indicating insufficient robustness, with a basic weight of 0.1.

[0025] Further weights are assigned based on the level of dissemination value. Exponentially growing elements, possessing breakthrough value and capable of rapidly generating widespread attention and subsequent research within the field, receive an additional weight of 0.3. Logically growing elements demonstrate steadily increasing academic value and sustained long-term influence, receiving an additional weight of 0.2. The final element weight is the sum of the basic weight and the additional weight. For example, an element with first-level robustness combined with exponentially growing dissemination value has a final weight of 0.4 plus 0.3 equaling 0.7; an element with second-level robustness combined with logically growing dissemination value has a final weight of 0.3 plus 0.2 equaling 0.5. By clarifying the scoring rules and quantitative weighting standards for each dimension, the importance of highly robust and highly dissemination-value elements is highlighted, ultimately determining the semantic element set.

[0026] Furthermore, the method provided in the application embodiments, which performs adversarial testing based on counterfactual trajectory perturbation and knowledge diffusion trajectory prediction, also includes: A counterfactual trajectory generator is employed to introduce subtle semantic perturbations. Based on the counterfactual trajectory judge, the conditional independence and element robustness among latent variables are measured to determine the first adversarial test data. The element robustness includes semantic, logical, and empirical dimensions. The semantic dimension characterizes the stability against textual perturbations, the logical dimension characterizes the fragility of the premise-conclusion chain, and the empirical dimension characterizes the reproducibility risk of the data. A knowledge predictor is then employed to perform knowledge diffusion trajectory prediction to determine the second diffusion test data based on diffusion patterns. The diffusion patterns are classified into exponential growth, logical growth, and impulsive decay patterns.

[0027] In this embodiment, a counterfactual trajectory generator is first employed to introduce subtle semantic perturbations. The counterfactual trajectory generator uses a large language model to parse the content of the target paper and rewrites the extracted semantic elements in a structured manner through cue word engineering and large language model fine-tuning techniques. Specifically, while maintaining the core proposition of the semantic elements, the counterfactual trajectory generator performs controlled adjustments to their non-core components. These adjustments include, but are not limited to, synonym replacement of key terms, word order transformation of sentence structure, and replacement of logical connectors. Through these operations, a set of counterfactual text variants are generated for each original semantic element, differing in expression details but maintaining consistency in core academic claims. These text variants generated through subtle semantic perturbations, together with the original semantic elements, constitute the counterfactual trajectory of that element.

[0028] Subsequently, the counterfactual trajectory determiner measures the conditional independence and robustness of latent variables to determine the first adversarial test data. The counterfactual trajectory determiner receives and processes the aforementioned counterfactual trajectories. In the semantic dimension evaluation, the counterfactual trajectory determiner quantifies the semantic distance between the original semantic element and each counterfactual text variant through semantic vector similarity calculation, thereby characterizing the element's stability against text perturbations; the smaller the range of semantic distance variation, the higher the stability. In the logical dimension evaluation, the counterfactual trajectory determiner analyzes, through natural language reasoning techniques, whether the logical implication relationship between the premise assertions and conclusion assertions implied by the semantic element is disrupted under the influence of perturbations, thereby characterizing the vulnerability of the premise-conclusion chain; the higher the degree of maintenance of the logical implication relationship, the lower the vulnerability. In the empirical dimension evaluation, the counterfactual trajectory determiner verifies the consistency and verifiability of the data, methods, or experimental conclusions cited in the semantic element under different perturbation versions by searching external structured academic databases and knowledge bases, thereby characterizing the data's reproducibility risk; the higher the verifiability, the lower the risk. The counterfactual trajectory judge integrates the evaluation results of the above three dimensions and infers the conditional independence by analyzing the change patterns of semantic representation latent variables under different perturbations. Finally, it generates a set of data containing multidimensional stability and independence measures for each semantic element, and defines the set of data as the first adversarial test data.

[0029] Simultaneously, a knowledge predictor is employed to perform knowledge diffusion trajectory prediction to determine the second diffusion test data based on diffusion patterns. Specifically, the knowledge predictor builds its predictive capabilities by training and analyzing citation networks and content evolution data of large-scale historical academic literature within the domain. First, the knowledge predictor uses a large language model to perform deep analysis of the target paper, extracting the core knowledge points corresponding to semantic elements. Subsequently, based on technologies such as graph neural networks, the knowledge predictor embeds these knowledge points into the constructed domain academic knowledge graph, simulating the process of them establishing connections with existing knowledge nodes as new nodes and diffusing over time. By analyzing the characteristics of the predicted influence accumulation curve, the knowledge predictor categorizes the propagation patterns of knowledge points into exponential growth patterns characterized by a rapid initial increase in influence, logical growth patterns characterized by a continuous, stable, and gradual accumulation of influence, or pulse decay patterns characterized by a rapid peak followed by a rapid decline in influence. Based on this classification result, the knowledge predictor assigns a diffusion pattern label to each semantic element; this label and its associated prediction criteria together constitute the second diffusion test data.

[0030] Step S200: Build and maintain the domain benchmark library.

[0031] In this embodiment, high-quality academic papers in the target field are first collected from an academic paper database. A large language model is then used to perform semantic parsing on the text, extracting semantic units such as innovative sentences, viewpoint sentences, research question sentences, concept definition sentences, and unresolved problem sentences. These are used to construct innovation benchmark libraries, viewpoint benchmark libraries, research question benchmark libraries, concept definition benchmark libraries, and unresolved problem benchmark libraries, respectively. The semantic units in each benchmark library are encoded using a semantic vectorization model and stored in a semantic vector database to form a computable semantic benchmark set. Subsequently, an incremental update mechanism is used to maintain each benchmark library, periodically introducing the latest paper corpus and combining it with manual review results for semantic retraining and data cleaning, ensuring the accuracy and timeliness of the benchmark library content.

[0032] Furthermore, the method provided in the application embodiments also includes: The semantic element set includes at least innovative sentences, viewpoint sentences, research question sentences, concept definition sentences, unresolved questions, and paper domain classifications; the domain benchmark library includes at least an innovative benchmark library, a viewpoint benchmark library, a research question benchmark library, a concept definition benchmark library, and an unresolved question benchmark library; wherein, the paper domain classification serves as the construction benchmark for the domain benchmark library.

[0033] In this embodiment, the semantic element set includes at least six semantic units: innovative sentences, viewpoint sentences, research question sentences, concept definition sentences, unresolved problem sentences, and paper domain classification. Specifically, innovative sentences embody the innovative content of the paper in terms of methodology or theory; viewpoint sentences express the author's academic insights and logical stance; research question sentences reveal the core research focus of the paper; concept definition sentences define the key terms and theoretical concepts in the paper; unresolved problem sentences reflect unsolved research challenges in the field; and paper domain classification is used to determine the disciplinary scope and research direction of the paper.

[0034] Based on the classification of paper domains as the construction benchmark, semantic elements within the same domain are clustered and integrated to establish innovation benchmark databases, viewpoint benchmark databases, research question benchmark databases, concept definition benchmark databases, and unsolved problem benchmark databases, so as to form a domain semantic reference system covering different semantic dimensions.

[0035] Step S300: Set the evaluation dimensions for quantifying the paper's value, perform multi-dimensional joint scoring based on the semantic element set and the domain benchmark library, and determine the paper's value score. The evaluation dimensions include at least the innovation point dimension, the author's viewpoint dimension, the research question dimension, and the concept definition dimension.

[0036] In this embodiment, the evaluation dimensions for quantifying the paper's value are first defined, including dimensions of innovation, author's viewpoint, research question, and concept definition. Then, a multi-dimensional joint scoring is performed based on the semantic element set and the domain benchmark library. This involves semantically matching the semantic elements in the target paper with the corresponding domain benchmark library, and quantifying each dimension using a preset scoring formula to obtain independent scores for each dimension. Finally, the scores corresponding to the innovation, author's viewpoint, research question, and concept definition dimensions are summed to obtain the paper's value score.

[0037] Furthermore, the method provided in the application embodiments also includes: The innovation dimension is calculated based on the number of innovative sentences and semantic novelty; the score calculation formula for the innovation dimension is as follows:

[0038] in, To score the innovation aspect, The target is the number of innovative sentences in the paper. The upper limit is set to 10. The average cosine similarity between all innovative sentences in the target paper and the innovative benchmark database. Mean cosine similarity is negatively correlated with semantic novelty.

[0039] In this embodiment, the innovation dimension is calculated based on the number of innovative sentences and semantic novelty. First, all innovative sentences are extracted from the semantic element set of the target paper, denoted as... To avoid imbalanced scoring due to an excessive number of innovative sentences in abnormal papers, a maximum of 10 innovative sentences is set. If the number of innovative sentences in the target paper exceeds this maximum, the maximum value will be used for calculation.

[0040] Next, each innovative sentence in the target paper is semantically matched with all innovative sentences in the innovation benchmark database, and the corresponding cosine similarity is calculated. Then, the average cosine similarity between all innovative sentences in the target paper and the innovation benchmark database is calculated. The lower the mean cosine similarity, the greater the semantic difference between the innovative sentence in the target paper and the existing innovative benchmark database, and the higher the semantic novelty.

[0041] Then, the score for the innovation dimension of the target paper is calculated using the scoring formula for innovation points. If the paper has no innovative sentences, then... Then the innovation dimension score If the number of innovative sentences reaches the upper limit and the semantic novelty is the highest (mean cosine similarity is 0), then 25 is the full score.

[0042] Furthermore, the method provided in the application embodiments also includes: The author's viewpoint dimension is calculated based on the semantic uniqueness and logical consistency of the viewpoint statement; the scoring formula for the author's viewpoint dimension is as follows:

[0043] in, Score the author's viewpoint dimension. The mean cosine similarity between all opinion sentences in the target paper and the opinion benchmark database. It is negatively correlated with semantic uniqueness; The average cosine similarity among all the viewpoint sentences in the target paper. It is positively correlated with logical consistency.

[0044] In this embodiment, the author's viewpoint dimension is calculated based on the semantic uniqueness and logical consistency of the viewpoint sentence. When calculating the score for the author's viewpoint dimension, the viewpoint sentence is first extracted directly from the semantic element set. Next, the extracted viewpoint sentence is matched against a viewpoint benchmark database, and their mean cosine similarity is calculated, denoted as [missing information]. Among them, the lower This indicates that the target paper's viewpoint has a greater semantic difference from existing viewpoints, demonstrating stronger semantic uniqueness.

[0045] Simultaneously, vector similarity calculations are performed on all pairwise argument sentences within the target paper, and their average internal cosine similarity is calculated, denoted as . This indicator reflects the degree of semantic consistency between opinion sentences; a higher value indicates a more coherent argumentative structure and stronger logical consistency.

[0046] Finally, the score for the author's viewpoint dimension is calculated using the formula for calculating the author's viewpoint dimension.

[0047] Furthermore, the method provided in the application embodiments also includes: The research question dimension is calculated based on the necessity and clarity of the research question; the scoring formula for the research question dimension is as follows:

[0048] in, To score the dimensions of the research question, The target paper studies the mean cosine similarity between problem sentences and an unsolved benchmark database. It is positively correlated with necessity; The mean cosine similarity between the research question sentences in the target paper and the research question benchmark database is used. It is negatively correlated with innovation; To study the perplexity of question sentences in the target paper, It is negatively correlated with clarity of expression; To study the average perplexity of question sentences; The weight distribution is used to characterize importance priority.

[0049] In this embodiment, the research question dimension is calculated by extracting the research question sentence from the semantic element set. This dimension uses the necessity and clarity of expression of the research question as its calculation logic, aiming to quantify the academic value and expression quality of the research question in the paper through multi-dimensional semantic comparison and perplexity analysis.

[0050] in, This is used to determine whether a research question sentence fits the unresolved problem benchmark library. In this application, unresolved problem sentences are used to establish the unresolved problem benchmark library.

[0051] First, the extracted research question sentences are semantically matched with an unresolved problem baseline database, and their mean cosine similarity is calculated, denoted as [missing information]. Higher The value indicates that the research question of the target paper has a high semantic similarity to the unsolved problems in the field, that is, the necessity of the research question is stronger.

[0052] Next, the research question sentences are compared with the research question benchmark database, and the mean cosine similarity is calculated, denoted as . Lower A higher value indicates that the research question is more different from existing questions, thus possessing greater innovativeness.

[0053] At the linguistic level, the complexity of the expression is measured by calculating the perplexity of the research question sentences in the target paper, denoted as... When calculating the perplexity of the research question sentence, firstly, the joint probability of the sentence is calculated based on the language model n-gram: for the sentence ( (The i-th word), calculate the conditional probability of the current word given the preceding words, and then multiply all conditional probabilities to obtain the joint probability of the sentence. The formula is: Next, we calculate the perplexity: to eliminate the influence of sentence length (n), we calculate the "nth root of the reciprocal of the joint probability", the formula is: .

[0054] In the embodiments of this application, It is the perplexity value of the research question sentence in a single paper within the target paper. It is the average perplexity of all papers in the research problem benchmark library.

[0055] Then, the score for the research question dimension is calculated using the scoring formula for the research question dimension. Among these, The weight distribution, which characterizes importance priority, is predefined. For example, (Prioritize necessity and solve problems first) (Innovation is secondary; avoid repetitive research.) (Ensuring clarity and delivering value).

[0056] Furthermore, the method provided in the application embodiments also includes: The concept definition dimension is calculated based on the accuracy, originality, and clarity of expression of the concept definition; the scoring formula for the concept definition dimension is as follows:

[0057] in, Define dimensions and scores for concepts. The average cosine similarity between the concept definition sentences in the target paper and the core layer of the concept definition benchmark library, where the core layer refers to the core consensus definitions that are invariant in the domain. It is positively correlated with accuracy; These are the weighting coefficients; The average cosine similarity between concept definition sentences in the target paper and the traditional layer of the concept definition benchmark library, where the traditional layer refers to classic expressions in the field in the past. It is negatively correlated with innovation; The perplexity of concept definition sentences in the target paper. It is negatively correlated with clarity of expression.

[0058] In this embodiment, the concept definition dimension uses the accuracy, originality, and clarity of expression of the concept definition as the calculation logic, aiming to quantify the scientific standardization and language expression quality of the concept definition in the paper through semantic matching and perplexity analysis.

[0059] First, concept definition sentences are extracted from the semantic element set. These sentences are used to describe the key concepts, variables, or theoretical categories of the research and are a fundamental component of the theoretical system in the paper. Then, the extracted concept definition sentences are semantically matched with a concept definition benchmark library (core layer), and their mean cosine similarity is calculated, denoted as . This metric measures the degree of consistency between concept definitions in a paper and the core definitions generally accepted in the field. A high similarity indicates that the paper's concept definitions are closer to mainstream academic standards, demonstrating stronger accuracy in the conceptual expression.

[0060] Next, the concept definition sentence is semantically compared with the concept definition benchmark library (traditional layer), and the mean cosine similarity is calculated, denoted as . The traditional benchmark library contains representative and inherited definitions from historical research in the field, used to measure whether a paper's concepts follow existing expression patterns. A low similarity indicates that the paper's concept definition differs significantly from traditional expressions, demonstrating high innovation; conversely, a high similarity indicates that the research concept is relatively conservative in terms of innovation.

[0061] At the linguistic level, the complexity of linguistic expression is measured by calculating the perplexity of concept definition sentences. This calculation process is identical to that used for calculating the perplexity of the research question sentences, thus obtaining the perplexity of concept definition sentences in the target paper. .

[0062] Based on the data obtained from the aforementioned calculations, the score for the concept definition dimension is calculated using the scoring formula for the concept definition dimension. .in, This is the weighting coefficient, defaulted to 0.6, to ensure accuracy takes precedence over innovativeness, conforming to academic norms; ensuring correct definition first, then considering innovation. The accuracy index is defined by... Characterization is positively correlated with the accuracy of the definition; the definition of innovativeness indicators is derived from... Representation is negatively correlated with innovativeness; the clarity of expression index is... Representation is negatively correlated with clarity of expression.

[0063] Furthermore, the method provided in the application embodiments also includes: The formula for calculating the paper's value score is as follows: If the semantic elements related to the first evaluation dimension are an empty set, it is determined that there is no valid data, and the dimension score of the first evaluation dimension is defined as 0. The first evaluation dimension is any one of the following evaluation dimensions: innovation point dimension, author's viewpoint dimension, research question dimension, and concept definition dimension. A similarity range is set. If the similarity calculation result of the evaluation dimension does not meet the similarity range, normalization processing based on the similarity range is performed. The similarity range is [0, 1].

[0064] In this embodiment of the application, the formula for calculating the paper value score is as follows: The innovation dimension score obtained from the aforementioned calculations is used to... Author's viewpoint dimension score Research Question Dimension Score and concept definition dimension score The summation yields the paper's value score.

[0065] If the semantic element set corresponding to the first evaluation dimension is empty, that is, if no semantic content related to this dimension can be identified in the paper text, such as missing sentences describing the innovation points, sentences expressing the author's viewpoints, sentences expressing research questions, or sentences defining concepts, then this dimension is judged to have no valid data.

[0066] An empty semantic element set means that this dimension lacks textual basis for calculation at the semantic level, making it impossible to analyze indicators such as similarity, perplexity, or consistency. To ensure the computational stability and comparability of the overall evaluation process, after determining that there is no valid data, the score of this dimension is directly defined as zero (0), indicating that this dimension does not participate in the calculation of the paper's overall score. The first evaluation dimension can be any one of the following: innovation point dimension, author's viewpoint dimension, research question dimension, or concept definition dimension.

[0067] To ensure the numerical standardization and comparability of semantic matching results across all evaluation dimensions, after calculating the similarity between the semantic element set and the benchmark database, a similarity range is set and the results are normalized. Specifically, the similarity range is first set to [0, 1], which limits the valid value range of the semantic similarity results. A similarity value of 0 indicates complete semantic irrelevance, while a value of 1 indicates complete semantic consistency, ensuring that semantic similarity has clear upper and lower boundaries on a numerical scale.

[0068] During the calculation process, the semantic element sets extracted from each evaluation dimension are used as the analysis objects. Sentences in these sets are semantically matched with the corresponding benchmark databases for each dimension. For example, sentences about innovation points in the innovation point dimension are matched with the innovation point benchmark database; sentences about opinions in the author's opinion dimension are matched with the author's opinion benchmark database; sentences about research questions in the research question dimension are matched with the research question benchmark database; and sentences about definition in the concept definition dimension are matched with the concept definition benchmark database. By calculating the cosine similarity between semantic vectors, the semantic closeness of each sentence to its corresponding sample sentences in the benchmark database is obtained.

[0069] Due to factors such as differences in corpus, text length, or errors in embedding vector calculation, some similarity results may exceed the theoretical range (e.g., values ​​slightly less than 0 or greater than 1), which can affect the accuracy of dimensional scores and the stability of the overall evaluation. Therefore, after obtaining the similarity calculation results, normalization processing based on the similarity range is performed on results that do not meet the set range [0, 1].

[0070] Normalization can be achieved by linear mapping or boundary truncation, adjusting all similarity results to the standard interval [0, 1] to maintain a uniform numerical distribution.

[0071] In summary, the embodiments of this application have at least the following technical effects: This application, based on a large language model, extracts a semantic element set from target papers. The extraction process includes first-order prompt word extraction and second-order testing and screening, with screening methods including adversarial testing and knowledge diffusion trajectory analysis. It also constructs and maintains a domain benchmark library; sets evaluation dimensions for quantifying paper value; and performs multi-dimensional joint scoring based on the semantic element set and the domain benchmark library to determine the paper's value score. The scoring is performed using dedicated scoring formulas for the dimensions of innovation, author viewpoints, research questions, and concept definitions. This invention addresses the technical problem of existing paper value evaluation relying on external citation data and being unable to directly quantify the paper's content. Through semantic element extraction and a multi-dimensional joint scoring mechanism, it achieves the technical effect of directly, objectively, and accurately quantifying the value of paper content.

[0072] Example 2, based on the same inventive concept as the paper value quantification evaluation method in the foregoing examples, such as... Figure 2 As shown, this application provides a device for quantifying the value of a paper. The device and method embodiments in this application are based on the same inventive concept. The device includes: The semantic element extraction module 11 is used to extract a set of semantic elements from the target paper based on a large language model. The extraction steps include first-order prompt word extraction and second-order test screening. The screening methods include adversarial testing and knowledge diffusion trajectory analysis. The benchmark library construction module 12 is used to construct and maintain a domain benchmark library. The scoring module 13 is used to set the evaluation dimensions for quantifying the paper's value, perform multi-dimensional joint scoring based on the semantic element set and the domain benchmark library, and determine the paper's value score. The scoring is calculated using dedicated scoring formulas for the dimensions of innovation, author's viewpoint, research question, and concept definition.

[0073] Furthermore, the device is also used to perform the following functions: Based on the first processing layer within the large language model, and using prompt word engineering, large language model fine-tuning, and preset extraction strategies, an initial set of semantic elements is extracted from the target paper. Based on the second processing layer within the large language model, adversarial testing based on counterfactual trajectory perturbation and knowledge diffusion trajectory prediction are performed to filter and weight the initial set of semantic elements, thereby determining the set of semantic elements.

[0074] Furthermore, the device is also used to perform the following functions: A counterfactual trajectory generator is employed to introduce subtle semantic perturbations. Based on the counterfactual trajectory judge, the conditional independence and element robustness among latent variables are measured to determine the first adversarial test data. The element robustness includes semantic, logical, and empirical dimensions. The semantic dimension characterizes the stability against textual perturbations, the logical dimension characterizes the fragility of the premise-conclusion chain, and the empirical dimension characterizes the reproducibility risk of the data. A knowledge predictor is then employed to perform knowledge diffusion trajectory prediction to determine the second diffusion test data based on diffusion patterns. The diffusion patterns are classified into exponential growth, logical growth, and impulsive decay patterns.

[0075] Furthermore, the device is also used to perform the following functions: The semantic element set includes at least innovative sentences, viewpoint sentences, research question sentences, concept definition sentences, unresolved questions, and paper domain classifications; the domain benchmark library includes at least an innovative benchmark library, a viewpoint benchmark library, a research question benchmark library, a concept definition benchmark library, and an unresolved question benchmark library; wherein, the paper domain classification serves as the construction benchmark for the domain benchmark library.

[0076] Furthermore, the device is also used to perform the following functions: The innovation dimension is calculated based on the number of innovative sentences and semantic novelty; the score calculation formula for the innovation dimension is as follows:

[0077] in, To score the innovation aspect, The target is the number of innovative sentences in the paper. The upper limit is set to 10. The average cosine similarity between all innovative sentences in the target paper and the innovative benchmark database. Mean cosine similarity is negatively correlated with semantic novelty.

[0078] Furthermore, the device is also used to perform the following functions: The author's viewpoint dimension is calculated based on the semantic uniqueness and logical consistency of the viewpoint statement; the scoring formula for the author's viewpoint dimension is as follows:

[0079] in, Score the author's viewpoint dimension. The mean cosine similarity between all opinion sentences in the target paper and the opinion benchmark database. It is negatively correlated with semantic uniqueness; The average cosine similarity among all the viewpoint sentences in the target paper. It is positively correlated with logical consistency.

[0080] Furthermore, the device is also used to perform the following functions: The research question dimension is calculated based on the necessity and clarity of the research question; the scoring formula for the research question dimension is as follows:

[0081] in, To score the dimensions of the research question, The target paper studies the mean cosine similarity between problem sentences and an unsolved benchmark database. It is positively correlated with necessity; The mean cosine similarity between the research question sentences in the target paper and the research question benchmark database is used. It is negatively correlated with innovation; To study the perplexity of question sentences in the target paper, It is negatively correlated with clarity of expression; To study the average perplexity of question sentences; The weight distribution is used to characterize importance priority.

[0082] Furthermore, the device is also used to perform the following functions: The concept definition dimension is calculated based on the accuracy, originality, and clarity of expression of the concept definition; the scoring formula for the concept definition dimension is as follows:

[0083] in, Define dimensions and scores for concepts. The average cosine similarity between the concept definition sentences in the target paper and the core layer of the concept definition benchmark library, where the core layer refers to the core consensus definitions that are invariant in the domain. It is positively correlated with accuracy; These are the weighting coefficients; The average cosine similarity between concept definition sentences in the target paper and the traditional layer of the concept definition benchmark library, where the traditional layer refers to classic expressions in the field in the past. It is negatively correlated with innovation; The perplexity of concept definition sentences in the target paper. It is negatively correlated with clarity of expression.

[0084] Furthermore, the device is also used to perform the following functions: The formula for calculating the paper's value score is as follows: If the semantic elements related to the first evaluation dimension are an empty set, it is determined that there is no valid data, and the dimension score of the first evaluation dimension is defined as 0. The first evaluation dimension is any one of the following: innovation point dimension, author's viewpoint dimension, research question dimension, and concept definition dimension. A similarity range is set. If the similarity calculation result of the evaluation dimension does not meet the similarity range, normalization processing based on the similarity range is performed. The similarity range is [0, 1].

[0085] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments of this specification. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some implementations, multitasking and parallel processing are possible or may be advantageous.

[0086] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0087] This specification and accompanying drawings are merely illustrative examples of this application and are intended to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from its scope. Therefore, if such modifications and modifications fall within the scope of this application and its equivalents, this application intends to include such modifications and modifications.

Claims

1. A method for quantitatively evaluating the value of a thesis, characterized in that, The method includes: Based on a large language model, a set of semantic elements is extracted from the target paper. The extraction steps include first-order prompt word extraction and second-order test screening. The screening methods include adversarial testing and knowledge diffusion trajectory analysis. Build and maintain a domain benchmark library; The evaluation dimensions for quantifying the value of a paper are set, and a multi-dimensional joint scoring based on the semantic element set and the domain benchmark library is performed to determine the paper's value score. Specifically, a dedicated scoring formula is used to calculate the score for the dimensions of innovation, author's viewpoint, research question, and concept definition. Extract a set of semantic elements from the target paper, including: Based on the first processing layer within the large language model, and using prompt word engineering, large language model fine-tuning, and preset extraction strategies, an initial set of semantic elements is extracted from the target paper. Based on the second processing layer within the large language model, adversarial testing based on counterfactual trajectory perturbation and knowledge diffusion trajectory prediction are performed to filter and weight the initial semantic element set, thereby determining the semantic element set. Perform adversarial testing based on counterfactual trajectory perturbations and knowledge diffusion trajectory prediction, including: A counterfactual trajectory generator is used to introduce small semantic perturbations. Based on the counterfactual trajectory judge, the conditional independence and element robustness between latent variables are measured to determine the first adversarial test data. The element robustness includes semantic dimension, logical dimension and empirical dimension. The semantic dimension represents the stability against text perturbations, the logical dimension represents the fragility of the premise-conclusion chain, and the empirical dimension represents the reproducibility risk of the data. A knowledge predictor is used to perform knowledge diffusion trajectory prediction to determine second diffusion test data based on diffusion patterns, wherein the diffusion patterns are divided into exponential growth, logical growth and impulsive decay patterns. The initial semantic element set is screened and weighted to determine the semantic element set, including: a first round of screening based on the first adversarial test data, clarifying the score determination rules for the semantic dimension, logical dimension, and empirical dimension, eliminating elements that do not reach the threshold in any dimension, and retaining high-resistance core elements that meet the requirements in all three dimensions; a second round of screening based on the second diffusion test data, eliminating elements with pulse decay diffusion patterns, as these elements have no long-term academic impact after short-term propagation, and retaining elements with exponential growth and logical growth diffusion patterns that can provide effective support for the development of the field; a layered approach is adopted in the weighting process. The weighted mechanism first calculates the total robustness score based on the scores of each dimension, then determines the robustness level and assigns basic weights based on the total score; next, it assigns additional weights to exponentially growing and logically growing elements according to their dissemination value level. Exponentially growing elements have breakthrough value and can quickly attract widespread attention and subsequent research in the field; logically growing elements show steady academic value and their long-term influence continues to increase; the final element weight is the sum of the basic weight and the additional weight. By clarifying the calculation rules for the scores of each dimension and the quantitative weighting standards, the importance of highly robust and highly dissemination value elements is highlighted, and the semantic element set is finally determined.

2. The method for quantitatively evaluating the value of a paper as described in claim 1, characterized in that, The semantic element set includes at least innovative sentences, opinion sentences, research question sentences, concept definition sentences, unresolved question sentences, and paper domain classifications; The domain benchmark library shall include at least an innovation benchmark library, a perspective benchmark library, a research question benchmark library, a concept definition benchmark library, and an unresolved problem benchmark library; The paper domain classification serves as the benchmark for constructing the domain benchmark library.

3. The method for quantitatively evaluating the value of a paper as described in claim 2, characterized in that, The innovation dimension is calculated based on the number of innovative sentences and semantic novelty. The scoring formula for the innovation dimension is as follows: in, To score the innovation aspect, The target is the number of innovative sentences in the paper. The upper limit is set to 10. The average cosine similarity between all innovative sentences in the target paper and the innovative benchmark database. Mean cosine similarity is negatively correlated with semantic novelty.

4. The method for quantitatively evaluating the value of a paper as described in claim 3, characterized in that, The author's viewpoint dimension is calculated based on the semantic uniqueness and logical consistency of the viewpoint sentence. The formula for calculating the score of the author's viewpoint dimension is as follows: in, Score the author's viewpoint dimension. The mean cosine similarity between all opinion sentences in the target paper and the opinion benchmark database. It is negatively correlated with semantic uniqueness; The average cosine similarity among all the viewpoint sentences in the target paper. It is positively correlated with logical consistency.

5. The method for quantitatively evaluating the value of a paper as described in claim 4, characterized in that, The dimension of research questions is calculated based on the necessity of the research question and the clarity of its expression. The formula for calculating the score of the research question dimension is as follows: in, To score the dimensions of the research question, The target paper studies the mean cosine similarity between problem sentences and an unsolved benchmark database. It is positively correlated with necessity; The mean cosine similarity between the research question sentences in the target paper and the research question benchmark database is used. It is negatively correlated with innovation; To study the perplexity of question sentences in the target paper, It is negatively correlated with clarity of expression; To study the average perplexity of question sentences; The weight distribution is used to characterize importance priority.

6. The method for quantitatively evaluating the value of a paper as described in claim 5, characterized in that, The concept definition dimension is calculated based on the accuracy, originality, and clarity of expression of the concept definition. The formula for calculating the score of the concept definition dimension is: in, Define dimensions and scores for concepts. The average cosine similarity between the concept definition sentences in the target paper and the core layer of the concept definition benchmark library, where the core layer refers to the core consensus definitions that are invariant in the domain. It is positively correlated with accuracy; These are the weighting coefficients; The average cosine similarity between concept definition sentences in the target paper and the traditional layer of the concept definition benchmark library, where the traditional layer refers to classic expressions in the field in the past. It is negatively correlated with innovation; The perplexity of concept definition sentences in the target paper. It is negatively correlated with clarity of expression.

7. The method for quantitatively evaluating the value of a paper as described in claim 6, characterized in that, The formula for calculating the paper's value score is as follows: ; If the semantic elements related to the first evaluation dimension are empty, it is determined that there is no valid data, and the dimension score of the first evaluation dimension is defined as 0. The first evaluation dimension is any one of the following evaluation dimensions: innovation point dimension, author's viewpoint dimension, research question dimension, and concept definition dimension. Herein, a similarity range is set. If the similarity calculation result of the evaluation dimension does not meet the similarity range, normalization processing based on the similarity range is performed, wherein the similarity range is [0, 1].

8. A device for quantifying the value of a thesis, characterized in that, The apparatus is used to perform a paper value quantification evaluation method as described in any one of claims 1-7, the apparatus comprising: The semantic element extraction module is used to extract a set of semantic elements from the target paper based on a large language model. The extraction steps include first-order prompt word extraction and second-order test screening. The screening methods include adversarial testing and knowledge diffusion trajectory analysis. The benchmark library construction module is used to build and maintain domain benchmark libraries; The scoring module is used to set the evaluation dimensions for quantifying the value of a paper, and to perform multi-dimensional joint scoring based on the semantic element set and the domain benchmark library to determine the paper's value score. Specifically, the scoring is calculated using dedicated scoring formulas for the dimensions of innovation, author's viewpoint, research question, and concept definition.

Images