Academic quotation prediction method and device, electronic equipment and readable storage medium

Abstract

The invention provides an academic quotation prediction method and device, electronic equipment and a readable storage medium. The method comprises the steps that query paper information corresponding to a current quotation prediction task is determined; the current quotation prediction task comprises a list-level quotation prediction task and a position-level quotation prediction task; based on a pre-constructed multi-level corpus, performing multi-level mixed retrieval according to the query paper information to obtain a target retrieval result; and based on a pre-trained generation model, according to the query paper information and the target retrieval result, generating a target quotation prediction result. The method is oriented to two citation prediction tasks with different granularities, multi-level mixed retrieval is carried out in a large-scale multi-level corpus according to query paper information to retrieve and enhance to generate a final citation prediction result, list-level citation modeling and position specific citation reasoning can be supported at the same time, and the accuracy of citation prediction is improved. The internal structure characteristics of the scientific text are better reserved, and accurate quotation prediction of the academic paper under different task granularities is realized.

Description

Technical Field

[0001] This invention relates to the field of academic information retrieval technology, and in particular to an academic citation prediction method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] With the rapid development of online academic publishing, the number of papers published each year is growing exponentially, forming a vast and ever-expanding online academic knowledge system. Citation networks, as the core link in the evolution of scientific knowledge, connect various research results through citation relationships. Citation prediction aims to predict previous research that a new paper will cite, which can facilitate the discovery of related work for individual researchers and reveal the hidden structures of knowledge dissemination in computational social sciences.

[0003] Existing methods mainly fall into three categories: First, retrieval methods based on semantic embedding, which rely solely on semantic similarity, making it difficult to model the complex relationships between citations, and their performance degrades in large-scale candidate sets; second, citation-aware text generation methods, which are mainly geared towards question-answering scenarios and lack support for academic document-level citations and position-sensitive citations; and third, existing citation prediction datasets and benchmarks, which generally suffer from problems such as task simplification, neglect of document hierarchy, crude method design, and imperfect evaluation systems.

[0004] In fact, academic citation relationships involve multiple dimensions such as methodological connections, temporal dependencies, and domain conventions. Moreover, scientific texts have multi-level structures, which urgently require more refined and structure-aware citation prediction methods. Summary of the Invention

[0005] This invention provides an academic citation prediction method, apparatus, electronic device, and readable storage medium to solve the problems of simplified task format and flattened prediction structure in existing citation prediction methods, and to achieve accurate citation prediction of academic papers at different task granularities.

[0006] On one hand, this invention provides an academic citation prediction method, comprising: determining the query paper information corresponding to the current citation prediction task; the current citation prediction task includes a list-level citation prediction task and a position-level citation prediction task, and the query paper information includes at least the title and abstract of the query paper; based on a pre-constructed multi-level corpus, performing a multi-level hybrid retrieval according to the query paper information to obtain target retrieval results; based on a pre-trained generative model, generating target citation prediction results according to the query paper information and the target retrieval results; the generative model is constructed based on a large language model and is obtained by training and optimizing it according to a list-level citation prediction sample set and a position-level citation prediction sample set.

[0007] Further, determining the query paper information corresponding to the current citation prediction task includes: when the current citation prediction task is a list-level citation prediction task, determining the title and abstract of the query paper as the query paper information; or, when the current citation prediction task is a position-level citation prediction task, determining the title, abstract, and text paragraphs of the query paper as the query paper information.

[0008] Furthermore, the multi-level corpus includes a first level, a second level, and a third level; the first level includes the field classification, title, and abstract of the academic paper; the second level includes the field classification, title, abstract, and introduction of the academic paper; the third level includes the full text of the academic paper excluding citations; the metadata of the first level, the second level, and the third level is consistent, and the metadata includes the author information, publication year, and publication location information of the academic paper.

[0009] Furthermore, the step of performing multi-level hybrid retrieval based on the query paper information, according to a pre-constructed multi-level corpus, to obtain the target retrieval results includes: encoding the query paper information into a query embedding vector; performing similarity searches at the first, second, and third levels of the multi-level corpus based on the query embedding vector, and returning the first-level retrieval results, the second-level retrieval results, and the third-level retrieval results; fusing the first-level retrieval results, the second-level retrieval results, and the third-level retrieval results using a reciprocal ranking, and calculating the fusion score of multiple candidate academic papers; re-ranking the multiple candidate academic papers according to the fusion score to obtain the target retrieval results; the target retrieval results include the titles, authors, publication years, and abstracts of the multiple retrieved papers.

[0010] Furthermore, based on the pre-trained retrieval device, similarity searches are performed at the first, second, and third levels of the multi-level corpus according to the query embedding vector, and the retrieval results at the first, second, and third levels are returned; wherein, the retrieval device is constructed based on a large language model and is fine-tuned by comparative learning based on query positive sample pairs and query negative sample pairs.

[0011] Furthermore, it also includes: evaluating the retrieval machine using recall and average inverse ranking; evaluating the list-level citation prediction task using recall, normalized depreciation cumulative gain, and hit rate; evaluating the position-level citation prediction task using position-aware citation accuracy; and evaluating the generative model using citation diversity entropy and illusion rate.

[0012] Furthermore, each sample in the list-level citation prediction sample set includes the title, abstract, and complete citation list of the queried paper sample; each sample in the position-level citation prediction sample set includes the title, abstract, text paragraph including citation placeholders, and the corresponding actual citation list of the queried paper sample.

[0013] Secondly, the present invention also provides an academic citation prediction device, comprising: a query paper information determination module, used to determine the query paper information corresponding to the current citation prediction task; the current citation prediction task includes a list-level citation prediction task and a position-level citation prediction task, and the query paper information includes at least the title and abstract of the query paper; a multi-level hybrid retrieval module, used to perform multi-level hybrid retrieval based on a pre-built multi-level corpus and the query paper information to obtain target retrieval results; and an academic citation prediction module, used to generate target citation prediction results based on a pre-trained generative model, according to the query paper information and the target retrieval results; the generative model is constructed based on a large language model and is obtained by training and optimization based on a list-level citation prediction sample set and a position-level citation prediction sample set.

[0014] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the academic citation prediction method as described above.

[0015] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the academic citation prediction method as described above.

[0016] The academic citation prediction method provided by this invention determines the query paper information corresponding to the current citation prediction task. The current citation prediction task includes list-level citation prediction and position-specific citation prediction tasks, and the query paper information includes at least the title and abstract of the query paper. Based on a pre-constructed multi-level corpus, a multi-level hybrid retrieval is performed according to the query paper information to obtain the target retrieval result. Based on a pre-trained generative model, a target citation prediction result is generated according to the query paper information and the target retrieval result. The generative model is built based on a large language model and is obtained by training and optimizing it based on list-level citation prediction sample sets and position-specific citation prediction sample sets. This method is designed for two different granularities of citation prediction tasks. By performing a multi-level hybrid retrieval in a large-scale multi-level corpus based on the query paper information, it enhances the generation of the final citation prediction result. It can simultaneously support list-level citation modeling and position-specific citation inference, better preserving the inherent structural characteristics of scientific texts and achieving accurate citation prediction of academic papers at different task granularities. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the academic citation prediction method provided in this embodiment of the invention.

[0019] Figure 2 This is a performance comparison chart of the single-level retrieval strategy and the multi-level fusion retrieval strategy provided in the embodiments of the present invention.

[0020] Figure 3 This is a citation quality assessment diagram provided in an embodiment of the present invention.

[0021] Figure 4 This is a cross-domain performance analysis diagram provided in an embodiment of the present invention.

[0022] Figure 5 This is a schematic diagram of the academic citation prediction device provided in an embodiment of the present invention.

[0023] Figure 6 This is a schematic diagram of the physical structure of the electronic device provided in the embodiment of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0025] Existing academic citation prediction methods mainly fall into three categories. The first category is embedding-based retrieval methods. Nils Reimers et al. proposed the Sentence-BERT method in their 2019 paper, "Sentence Embeddings using SiameseBERT-Networks" (published at the EMNLP conference), which learns sentence embeddings through Siamese networks. Tianyu Gao et al. proposed the SimCSE method in their 2021 paper, "SimCSE: Simple Contrastive Learning of Sentence Embeddings" (published at the EMNLP conference), which uses contrastive learning to enhance semantic representations. A common drawback of these methods is that they only focus on semantic similarity, failing to capture the complex relationships within academic citations, and their retrieval performance significantly decreases as the candidate set expands.

[0026] The second category is citation-aware text generation methods. The 2022 paper "Teaching Language Models to Support Answers with Verified Quotes" by Jacob Menick et al. and the 2023 paper "Enabling Large Language Models to Generate Text with Citations" by TianyuGao et al. (published at the EMNLP conference) explored citation-aware text generation. However, these methods are primarily designed for online question-answering scenarios and lack specific support for academic citation prediction tasks, failing to handle document-level citation modeling and position-specific citation inference.

[0027] The third category is citation prediction datasets and benchmarks. Arman Cohan et al.'s 2020 paper "SPECTER: Document-level Representation Learning using Citation-informed Transformers" (published at the ACL conference) constructed the SciDocs dataset, and Tarek Saier et al.'s 2023 paper "unarXive 2022: All arXiv Publications Pre-Processed for NLP" constructed the unarXive dataset. However, existing benchmarks suffer from four main shortcomings: First, the task is oversimplified, with existing benchmarks typically treating citation prediction as a binary classification or document-level retrieval problem within a fixed corpus, thus becoming detached from real-world scenarios. Second, they neglect the hierarchical structure of scientific literature, often flattening complex hierarchical organizations into textual knowledge graphs, ignoring the multi-level organization of scientific discourse. Third, the method design is suboptimal, with most existing methods relying on fine-tuning simple embedding-based retrievers, making it difficult to capture the subtle structural characteristics of citation relationships. Fourth, there is a lack of a systematic evaluation framework, with existing benchmarks typically focusing only on citation network construction and simple embedding-based methods, failing to provide a comprehensive evaluation of more advanced methods such as retrieval enhancement generation.

[0028] Academic citation relationships are multi-dimensionally complex, encompassing methodological relevance, chronological order, and domain-specific conventions, characteristics that extend beyond simple semantic similarity. Furthermore, scientific texts possess inherent hierarchical structures and rhetorical organization, with different granularities of textual information contributing differently to citation prediction. Therefore, there is an urgent need for an academic citation prediction method that can simultaneously address issues such as task granularity differentiation, hierarchical corpus construction, retrieval method optimization, and system evaluation frameworks, in order to fully realize its potential in academic literature recommendation.

[0029] In view of this, the present invention proposes an academic citation prediction method based on retrieval enhancement generation, specifically, Figure 1 A flowchart illustrating the academic citation prediction method provided in an embodiment of the present invention is shown.

[0030] like Figure 1As shown, the method includes: S110, determining the query paper information corresponding to the current citation prediction task; the current citation prediction task includes list-level citation prediction tasks and position-level citation prediction tasks, and the query paper information includes at least the title and abstract of the query paper; S120, based on a pre-built multi-level corpus, performing multi-level hybrid retrieval according to the query paper information to obtain target retrieval results; S130, based on a pre-trained generative model, generating target citation prediction results according to the query paper information and the target retrieval results; the generative model is built based on a large language model and is obtained by training and optimizing according to list-level citation prediction sample sets and position-level citation prediction sample sets.

[0031] The following will provide a detailed description of steps S110-S130 and related steps.

[0032] S110, determine the query paper information corresponding to the current citation prediction task; the current citation prediction task includes list-level citation prediction tasks and position-level citation prediction tasks, and the query paper information includes at least the title and abstract of the query paper.

[0033] It is easy to understand that the academic citation prediction method provided in this embodiment is designed for two different granularities of citation prediction tasks: list-level citation prediction and position-level citation prediction. List-level citation prediction focuses on coarse-grained, list-specific predictions, while position-level citation prediction focuses on fine-grained, position-specific predictions. This dual-granularity citation prediction task design is more comprehensive and better suited to real-world academic collaboration scenarios, because researchers need to find relevant literature for the entire paper (list-level) and also find precise citations for specific paragraphs (position-level), overcoming the problem of simplistic task formats in existing citation prediction methods.

[0034] List-level citation prediction tasks aim to generate a sorted list of complete citations (of the query paper) based on given query paper information, while position-level citation prediction tasks aim to generate a sorted list of citations (of specific text paragraphs in the query paper) based on given query paper information. The query paper information used in the two tasks is different.

[0035] Specifically, if the current citation prediction task is a list-level citation prediction task, the corresponding query paper information includes the title and abstract of the query paper; if the current citation prediction task is a position-level citation prediction task, the corresponding query paper information includes the title, abstract, and specific text paragraphs that need to be citation predicted. The current citation prediction task is determined based on the researcher's actual needs and is not specifically limited here.

[0036] It should be noted that the papers searched can be either publicly available papers or legally used unpublished papers, and no specific limitation is made here.

[0037] Based on the determination of the queried paper information corresponding to the current citation prediction task in step S110, step S120 is further executed.

[0038] S120: Based on a pre-built multi-level corpus, perform multi-level hybrid retrieval according to the queried paper information to obtain the target retrieval results.

[0039] The pre-built multi-level corpus includes a large number of academic papers, spanning, but not limited to, six major fields: computer science, mathematics, physics, quantitative finance, quantitative biology, and statistics. This hierarchical design allows the system to flexibly access information of different granularities according to different task requirements, and compared with existing flat knowledge graph methods, it can better preserve the inherent structural characteristics of scientific texts. The "multi-level" in the multi-level corpus is at least two, but no specific limitation is made here.

[0040] For example, in one specific embodiment, the multi-level corpus is a three-level corpus. The first level includes the subject classification, title, and abstract of the academic paper for rapid relevance / topic assessment. The second level adds the introduction to the first level, including the subject classification, title, abstract, and introduction of the academic paper, to achieve a deeper understanding of the research motivation and methodology. The third level includes the full text of the academic paper (excluding citations) and the conclusion, to provide comprehensive content coverage. The multiple levels of the multi-level corpus have consistent metadata, including at least the author information, publication year, and publication location of the academic paper.

[0041] Based on the known information of the query papers corresponding to the current citation prediction task, similarity searches / retrievals are performed at each level of the multi-level corpus according to the query paper information, and the retrieval results for each level are returned. The target retrieval result is obtained by fusing the retrieval results from all levels. The fusion algorithm used here can be a reciprocal ranking fusion algorithm or other fusion algorithms; no specific limitation is made here. The target retrieval result includes the titles, authors, publication years, and abstracts of multiple relevant retrieved papers (denoted as retrieved papers).

[0042] The multi-level hybrid retrieval strategy adopted in this embodiment can enhance retrieval diversity while maintaining robustness under different corpus availability conditions.

[0043] In step S120, based on the pre-built multi-level corpus, a multi-level mixed search is performed according to the queried paper information to obtain the target search results. Then, step S130 is further executed.

[0044] S130, Based on a pre-trained generative model, a target citation prediction result is generated according to the queried paper information and the target retrieval result; the generative model is built based on a large language model and is obtained by training and optimization based on a list-level citation prediction sample set and a position-level citation prediction sample set.

[0045] The generative model is built on large language models (such as Qwen3-4B and Qwen3-Coder-30BA3B models) and is obtained by training and optimization on list-level citation prediction sample sets and position-level citation prediction sample sets. The specific training and optimization process can support four experimental settings: few-shot context learning, retrieval-enhanced generation, supervised fine-tuning, and a combination of supervised fine-tuning and retrieval-enhanced generation.

[0046] Each sample in the list-level citation prediction sample set includes the title, abstract, and complete citation list of the query paper sample. Each sample in the position-level citation prediction sample set includes the title, abstract, text paragraph including citation placeholders, and the corresponding actual citation list of the query paper sample. The list-level and position-level citation prediction sample sets can be obtained by preprocessing academic papers from multi-level corpora, or by collecting and preprocessing papers from other publicly available sources outside of multi-level corpora; no specific limitation is made here.

[0047] It should be noted that the intersection of the list-level citation prediction sample set and the position-level citation prediction sample set with academic papers in the multi-level corpus is empty.

[0048] In actual citation prediction, the query paper information determined in step S110 and the target retrieval results obtained in step S120, along with task requirement prompts (such as specifying the input and output formats), are input into the trained generative model to generate the target citation prediction results. The target citation prediction results include at least the titles of one or more relevant academic papers in a sorted order.

[0049] In this embodiment, the query paper information corresponding to the current citation prediction task is determined. The current citation prediction task includes list-level citation prediction and position-level citation prediction tasks, and the query paper information includes at least the title and abstract of the query paper. Based on a pre-built multi-level corpus, a multi-level hybrid retrieval is performed according to the query paper information to obtain the target retrieval result. Based on a pre-trained generative model, a target citation prediction result is generated according to the query paper information and the target retrieval result. The generative model is built based on a large language model and is obtained by training and optimizing it according to the list-level citation prediction sample set and the position-level citation prediction sample set. This method is designed for two citation prediction tasks with different granularities. By performing a multi-level hybrid retrieval in a large-scale multi-level corpus according to the query paper information, the final citation prediction result is generated through retrieval enhancement. It can simultaneously support list-level citation modeling and position-specific citation inference, better preserve the inherent structural characteristics of scientific texts, and achieve accurate citation prediction of academic papers at different task granularities.

[0050] Based on the above embodiments, the following will further describe in detail the construction process of the multi-level corpus in step S120.

[0051] In this embodiment, the multi-level corpus is a three-level corpus.

[0052] Specifically, firstly, an incremental pipeline is used to systematically collect academic papers from the target field over the past ten years from Google Scholar, and an adaptive query strategy is implemented: For field categories with fewer than 2,000 academic papers, due to the low frequency of document updates and the small total number of papers, a year-based filtering method is used for batch collection. This method reduces the number of queries and avoids redundant operations caused by excessive segmentation. For field categories with 2,000 to 12,000 academic papers, due to the relatively active output of documents in this category, a monthly query strategy is adopted to maintain a reasonable request frequency while ensuring data timeliness. For field categories with more than 12,000 papers, which are usually high-output fields (such as artificial intelligence and biomedicine) with rapid document growth, a fine-grained weekly collection method is adopted to more accurately capture the latest research results and prevent the large amount of data in a single query from affecting system stability.

[0053] It should be noted that the target field can be any or a combination of fields such as computer science, mathematics, physics, quantitative finance, quantitative biology, and statistics, without any specific limitations.

[0054] For example, in one specific embodiment, the target field is computer science. This field has a large number of papers, and a weekly collection strategy was adopted to prevent the loss of API-related data. Ultimately, approximately 250,000 papers in the computer science field were collected, covering multiple subfields such as artificial intelligence, machine learning, computer vision, natural language processing, data mining, and software engineering.

[0055] For example, in another specific embodiment, the target domain covers computer science, mathematics, physics, quantitative finance, quantitative biology, and statistics. Papers are extracted proportionally from each domain to ensure that the test set covers all major domains. The final multi-level corpus contains 554,719 papers, including approximately 250,000 in computer science, 120,000 in mathematics, 100,000 in physics, 30,000 in quantitative finance, 40,000 in quantitative biology, and 35,000 in statistics.

[0056] Subsequently, a three-tiered structure was constructed. The first tier includes the academic paper's domain classification, title, and abstract, with an average of 200 to 400 tokens per paper for rapid relevance assessment. The second tier adds an introduction to the first tier, increasing the number of tokens by an average of 600 to 1000 per paper, enabling a deeper understanding of the research motivation and methodology. The third tier contains the full text of the academic paper (excluding citations) and the conclusion, with an average of 4000 to 8000 tokens per paper, providing comprehensive content coverage. All tiers maintain consistent metadata, including author information, publication year, and publication location.

[0057] Based on the above, a multi-level corpus for subsequent multi-level hybrid retrieval can be constructed. Compared with the flat knowledge graph method, this hierarchical design better preserves the inherent structural characteristics of scientific texts.

[0058] Based on the above embodiments, the multi-level hybrid retrieval process in step S120 will be described in detail below.

[0059] Based on a pre-built multi-level corpus, a multi-level hybrid retrieval is performed according to the query paper information to obtain the target retrieval results. This includes: encoding the query paper information into a query embedding vector; performing similarity searches at the first, second, and third levels of the multi-level corpus based on the query embedding vector, and returning the first-level, second-level, and third-level retrieval results respectively; fusing the first-level, second-level, and third-level retrieval results using a reciprocal ranking, and calculating the fusion score of multiple candidate academic papers; re-ranking the multiple candidate academic papers according to the fusion score to obtain the target retrieval results; the target retrieval results include the titles, authors, publication years, and abstracts of multiple retrieved papers.

[0060] The process is straightforward: first, the query paper information is encoded into a query embedding vector. Then, the retrieval engine calculates the cosine similarity between the query embedding vector and the academic paper embeddings in the first, second, and third levels of the multi-level corpus. The system then returns the top N academic papers in each level, ranked by cosine similarity (from highest to lowest), as the retrieval results for that level. Here, N is a positive integer and can be set according to actual needs, such as 20; no specific limitation is made here.

[0061] Subsequently, the results from the first, second, and third levels of search were merged using a reciprocal ranking method to calculate the fusion score for multiple candidate academic papers. For each candidate academic paper in the first / second / third level search results, its fusion score is calculated as follows (1).

[0062] (1).

[0063] In equation (1), Indicates candidate academic papers In the The cosine similarity ranking in the hierarchical search results. If a candidate academic paper does not appear in the top N academic papers list of a certain level, then the contribution of that level is 0.

[0064] Finally, based on the fusion score of all candidate academic papers The papers are re-sorted from largest to smallest to obtain the top M papers, which will be used to generate citation prediction results later. Here, M is a positive integer less than or equal to N, and can be set according to actual needs, such as 10, but is not limited here.

[0065] The retrieval system used for multi-level hybrid retrieval employs Qwen3-Embedding-8B as its base model, fine-tuned using the Swift framework, with a training configuration of a learning rate of 5e-5, a batch size of 16, and a maximum sequence length of 4608. 30,000 query paper samples and their cited references are selected from the multi-level corpus, and 400,000 positive and negative query sample pairs are constructed for comparative learning and training.

[0066] Specifically, 30,000 query papers in the target domain (e.g., computer science) are selected from a multi-level corpus, and the references cited in each paper are extracted. For each query paper sample... On average, 13 to 15 positive samples were extracted from each level of the multi-level corpus. (The cited paper) constructed approximately 400,000 positive query pairs. The extraction process uses the title and abstract of the query paper sample, representing positive samples. The text representation is the same as the text representation used at the corresponding corpus level, for example, the positive samples extracted from the first level. Represented by domain category, title, and abstract.

[0067] For each sample of queried papers Two strategies were used to construct the negative sample set. The first method uses near-nearest neighbor negative samples, employing the FAISS index to retrieve and query paper samples from a multi-level corpus. Semantically similar but uncited academic papers, sample papers for each query. Five near-nearest neighbor negative samples are sampled. The second method is intra-batch negative sampling, which uses positive samples from other queried papers in the same batch as the current queried paper sample. negative samples When the batch size is 16, each queried paper sample Fifteen negative samples were obtained within a batch.

[0068] After constructing a good positive sample pair of query papers and query negative samples of papers Based on this, the InfoNCE loss with temperature scaling is used for training, as shown in equation (2).

[0069] (2).

[0070] In equation (2), , This is a temperature parameter, and its value can be 0.05.

[0071] Training is performed for 5 epochs, each lasting approximately 3 hours, for a total training time of approximately 15 hours. The training progress is monitored using a validation set, and training is stopped early when the MRR@50 on the validation set fails to improve for two consecutive epochs. This process yields a finely tuned retrieval system.

[0072] Based on the above embodiments, the following will further describe in detail the construction process of the list-level citation prediction sample set and the position-level citation prediction sample set in step S130.

[0073] Regarding the construction of the list-level citation prediction sample set: Papers with standard citation formats were selected from academic papers in the target field (e.g., computer science) from a multi-level corpus. Non-standard cases such as duplicate citations and citation errors were excluded. Papers with at least 10 citations and no more than 100 citations were retained. Finally, a list-level citation prediction sample set containing 5200 instances (samples) was constructed. Each instance includes the title, abstract, and complete citation list of the queried paper sample.

[0074] Regarding the construction of the position-level citation prediction sample set: A multi-stage approach was used to create fine-grained citation scenarios for academic papers in the target domain from a multi-level corpus: After initial filtering of non-standard citations, the citation distribution pattern of each academic paper was analyzed to identify the three most frequently cited parts (usually the introduction, related work, and methods sections); for these highly cited parts, the top three most frequent citations with valid identifiers were retained, and other citation markers were removed, creating a clean text context with precisely located citation placeholders [ref]; papers with any single citation appearing more than ten times in a section were excluded to prevent citation bias. Finally, a position-level citation prediction sample set containing 3500 instances was constructed, each instance containing the title, abstract, text paragraph containing citation placeholders, and corresponding actual citations of the query paper sample.

[0075] To ensure rigorous evaluation integrity, all academic papers that were successfully processed and included in the list-level and position-level citation prediction sample sets were permanently removed from the original multi-level corpus to prevent data leakage.

[0076] After constructing the list-level citation prediction sample set and the position-level citation prediction sample set, the generative model is trained using the list-level citation prediction sample set and the position-level citation prediction sample set. The training process can support four experimental settings: few-shot context learning, retrieval-enhanced generation, supervised fine-tuning, and a combination of supervised fine-tuning and retrieval-enhanced generation.

[0077] For few-shot context learning experiments, the generative model can be a closed-source model such as GPT-5, Claude-4-Sonnet, or Gemini-2.5-Pro, or an open-source model such as Kimi-K2 or Deepseek-v3, and inference can be performed via API calls. The generation temperature is set to 0.1, the presence penalty to 1.0 (if the API supports it), and the maximum number of generated tokens is 2048. The model generates prediction results in JSON format based on the prompt words, including a list of cited titles and inference explanations.

[0078] For supervised fine-tuning experiments, the generative model used Qwen3-4B and Qwen3-Coder-30BA3B as the base models, fine-tuned on list-level and position-level citation prediction sample sets (taking query paper sample information as input, the predicted citation list as output, and the difference between the predicted citation list and the actual citations as the training loss). Distributed training was performed using the Megatron-LM framework with 32 NVIDIA A100 GPUs, a learning rate of 7e-6, a batch size of 64, and a maximum sequence length of 131072 to support long contextual inputs. Full fine-tuning was used instead of parameter-efficient fine-tuning to fully adapt to the characteristics of the academic citation prediction task. Training lasted for 3 epochs, each approximately 0.76 hours, for a total training time of approximately 2.28 hours.

[0079] For the experiment combining supervised fine-tuning and retrieval enhancement generation, supervised fine-tuning was performed first, and then the retrieval results were integrated during inference.

[0080] Based on the above, a finely tuned generative model can be obtained. In actual citation prediction, a multi-level hybrid retrieval is performed on the given query paper information to obtain the top-10 relevant academic papers. The target retrieval results and the query paper information are then input into the finely tuned generative model, which outputs the final target citation prediction results, including a list of predicted citation titles and inference explanations.

[0081] Based on the above embodiments, the following will further describe in detail the evaluation process of the retrieval engine, the generation model, and the citation prediction tasks at two different granularities.

[0082] The retrieval system is evaluated using recall and inverse average ranking; the list-level citation prediction task is evaluated using recall, normalized depreciation cumulative gain, and hit rate; the position-aware citation accuracy is evaluated using position-level citation prediction; and the generative model is evaluated using citation diversity entropy and illusion rate.

[0083] Specifically, the retrieval evaluation uses two standard information retrieval metrics: Recall@k (k-value recall rate) and MRR@k (mean reciprocal rank of k-value). Recall@k emphasizes the proportion of relevant items recalled out of all relevant items in the top k results. The reciprocal rank in MRR@k refers to the reciprocal rank of the first relevant result.

[0084] The evaluation of list-level citation prediction tasks uses Recall@k (k-value recall), NDCG@k (k-value normalized depreciation cumulative gain), and Hit@k (k-value hit rate). In NDCG@k, depreciation reflects the smaller the contribution of items ranked later in the list. Hit@k considers a hit as long as there is at least one relevant item among the first k results.

[0085] The position-level citation prediction task evaluation uses the position-aware citation accuracy PACA@k, which considers both correctness and ranking position, as shown in equation (3).

[0086] (3).

[0087] In equation (3), This refers to the total number of reference placeholders. This represents the correct predictions within the top-k range. Indicates a correct prediction The location.

[0088] Generative model evaluation employs citation diversity entropy (CDE) and illusion rate. CDE measures the diversity of predicted citations across research subcategories within each major domain. have Each subcategory, set Representation field subcategories The proportion of Chinese citations Indicates the field The total proportion of citations. The CDE expression is as follows (4).

[0089] (4).

[0090] The illusion rate quantifies the proportion of citations pointing to non-existent papers. For a predicted citation list P, let V represent the set of papers that can be verified in the real world through a bibliographic database. Illusion Rate The expression is as follows (5).

[0091] (5).

[0092] In equation (5), The lower illusion rate indicates that the predictions cannot be verified as genuine papers, suggesting a better model foundation and reduced generation of fictitious citations.

[0093] In one specific embodiment, for list-level citation prediction tasks, Recall@20, Recall@40, NDCG@20, NDCG@40, Hit@20, and Hit@40 are calculated. For position-level citation prediction tasks, PACA@10, PACA@20, and PACA@40 are calculated. For retrieval-only evaluation, Recall@20, Recall@50, MRR@20, and MRR@50 are calculated. Furthermore, the citation diversity entropy (CDE) and illusion rate (Halluc.) are calculated for the generative model.

[0094] In this embodiment, on a list-level citation prediction test set in the field of computer science, the retrieval machine achieved a Recall@20 of 0.0982 and an MRR@20 of 0.3215. With a fully configured generative model (supervised fine-tuning plus retrieval-enhanced generation), it achieved a Recall@20 of 0.078, an NDCG@20 of 0.371, a Hit@20 of 0.195, a citation diversity entropy of 3.21, and an illusion rate of 4.7%. On a position-level citation prediction test set, it achieved PACA@10 of 0.285, PACA@20 of 0.311, and PACA@40 of 0.349.

[0095] In some other embodiments, Figure 2 The diagram shows a performance comparison between the single-level retrieval strategy and the multi-level fusion retrieval strategy provided in the embodiments of the present invention, demonstrating that multi-level fusion outperforms single-level retrieval across all embedding models.

[0096] exist Figure 2 The text discusses four models: BGE-M3, M-E5-Large, Qwen3-E-8B, and CitationRetriever-8B (the retrieval tool provided in this invention). The horizontal axis includes two sets of comparisons, and the vertical axis represents the performance metric MRR@50. Level 1 (brown) represents a single-level retrieval strategy using only the title and abstract (first level); Level 2 (purple) represents a single-level retrieval strategy including the introduction (second level); Level 3 (green) represents a single-level retrieval strategy including the full text (excluding citations) (third level); and Fused (blue) represents a hybrid retrieval strategy incorporating information from multiple levels.

[0097] according to Figure 2 It can be seen that the hybrid retrieval strategy corresponding to "Fused" achieved the highest score among all models, indicating that integrating text information of different granularities (such as paragraph level, sentence level, and document level) is crucial for citation inference, and multi-level fusion is superior to single-level retrieval in all embedding models.

[0098] Figure 3 The diagram illustrates the citation quality assessment provided by an embodiment of the present invention, showcasing the citation diversity entropy and hallucination rate across models. Citation diversity entropy (CDE) measures the diversity of citation selection, while the hallucination rate measures the proportion of generated false or non-existent citations.

[0099] exist Figure 3The invention involves four models: CitationGenerator-30B (brown, the generative model provided by this invention), Gemini-2.5-Pro ​​(purple), GPT-5 (green), and Claude-4-Sonnet (blue).

[0100] according to Figure 3 As can be seen, in the citation diversity entropy comparison on the left, CitationGenerator-30B has the highest CDE, indicating that it has the strongest citation coverage and diversity under RAG (Retrieval-Augmented Generation) support. In the illusion rate comparison on the right, CitationGenerator-30B has the lowest illusion rate (only 4.9%), which is far lower than other models.

[0101] Figure 4 The diagram illustrates the cross-domain performance analysis provided by an embodiment of the present invention, showing the performance of the generative model in six academic domains. It is used to compare the citation prediction performance of different large language models in multiple academic domains. The evaluation index is NDCG@20, which is "the ranking quality of relevant citations in the top 20 prediction results".

[0102] exist Figure 4 The data involves Gemini-2.5-Pro ​​(green circle), GPT-5 (red square), Claude-4-Sonnet (purple triangle), and CitationGenerator-30B (blue cross, the generative model provided by this valve). The horizontal axis represents different academic fields, including All (comprehensive across all fields), CS (Computer Science), Math, Physics, QuantFin (Quantitative Finance), QuantBio (Quantitative Biology), and Stats (Statistics). The vertical axis represents the performance metric NDCG@20.

[0103] according to Figure 4 As can be seen, the NDCG@20 of all models in the "All" category is between 0.30 and 0.37, with CitationGenerator-30B performing best (approximately 0.36), slightly higher than the other models. Within a single domain category, CitationGenerator-30B performs best in most domains, especially significantly outperforming in CS, Physics, and QuantBio.

[0104] Corresponding to the academic citation prediction methods described in the above embodiments, the present invention also proposes an academic citation prediction device.

[0105] Specifically, Figure 5A schematic diagram of the academic citation prediction device provided in an embodiment of the present invention is shown.

[0106] like Figure 5 As shown, the device includes: a query paper information determination module 510, used to determine the query paper information corresponding to the current citation prediction task; the current citation prediction task includes a list-level citation prediction task and a position-level citation prediction task, and the query paper information includes at least the title and abstract of the query paper; a multi-level hybrid retrieval module 520, used to perform multi-level hybrid retrieval based on a pre-built multi-level corpus and the query paper information to obtain target retrieval results; and an academic citation prediction module 530, used to generate target citation prediction results based on a pre-trained generative model, according to the query paper information and the target retrieval results; the generative model is built based on a large language model and is obtained by training and optimization based on a list-level citation prediction sample set and a position-level citation prediction sample set.

[0107] In this embodiment, the query paper information determination module 510 determines the query paper information corresponding to the current citation prediction task. The current citation prediction task includes list-level citation prediction tasks and position-level citation prediction tasks. The query paper information includes at least the title and abstract of the query paper. The multi-level hybrid retrieval module 520 performs multi-level hybrid retrieval based on a pre-built multi-level corpus and obtains the target retrieval result based on the query paper information. The academic citation prediction module 530 generates the target citation prediction result based on a pre-trained generative model and the query paper information and the target retrieval result. The generative model is built based on a large language model and is obtained by training and optimizing it based on list-level citation prediction sample sets and position-level citation prediction sample sets. This device is designed for two different granularity citation prediction tasks. By performing multi-level hybrid retrieval in a large-scale multi-level corpus based on the query paper information, it enhances the generation of the final citation prediction result. It can simultaneously support list-level citation modeling and position-specific citation inference, better preserve the inherent structural characteristics of scientific texts, and achieve accurate citation prediction of academic papers at different task granularities.

[0108] It should be noted that the academic citation prediction device provided in the embodiments of the present invention can be referred to in correspondence with the academic citation prediction methods described in the above embodiments, and will not be repeated here.

[0109] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute an academic citation prediction method. This method includes: determining the query paper information corresponding to the current citation prediction task; the current citation prediction task includes a list-level citation prediction task and a position-level citation prediction task, and the query paper information includes at least the title and abstract of the query paper; based on a pre-built multi-level corpus, performing a multi-level hybrid retrieval based on the query paper information to obtain target retrieval results; and based on a pre-trained generative model, generating target citation prediction results based on the query paper information and the target retrieval results; the generative model is built based on a large language model and is obtained by training and optimization based on a list-level citation prediction sample set and a position-level citation prediction sample set.

[0110] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0111] On the other hand, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the academic citation prediction method provided by the above methods. This method includes: determining query paper information corresponding to the current citation prediction task; the current citation prediction task includes a list-level citation prediction task and a position-level citation prediction task, and the query paper information includes at least the title and abstract of the query paper; performing a multi-level hybrid retrieval based on a pre-built multi-level corpus and the query paper information to obtain target retrieval results; and generating target citation prediction results based on a pre-trained generative model, according to the query paper information and the target retrieval results; the generative model is constructed based on a large language model and is obtained by training and optimization using a list-level citation prediction sample set and a position-level citation prediction sample set.

[0112] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0113] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An academic citation prediction method, characterized in that, include: Determine the query paper information corresponding to the current citation prediction task; The current citation prediction task includes list-level citation prediction tasks and position-level citation prediction tasks, and the queried paper information includes at least the title and abstract of the queried paper. Based on a pre-built multi-level corpus, a multi-level hybrid retrieval is performed according to the queried paper information to obtain the target retrieval results; Based on a pre-trained generative model, target citation prediction results are generated according to the queried paper information and the target retrieval results. The generative model is built on a large language model and is obtained by training and optimization based on list-level citation prediction sample sets and position-level citation prediction sample sets.

2. The academic citation prediction method according to claim 1, characterized in that, The process of determining the query paper information corresponding to the current citation prediction task includes: When the current citation prediction task is a list-level citation prediction task, the title and abstract of the queried paper are determined as the queried paper information; or, When the current citation prediction task is a position-level citation prediction task, the title, abstract, and text paragraphs of the queried paper are determined as the queried paper information.

3. The academic citation prediction method according to claim 1, characterized in that, The multi-level corpus includes a first level, a second level, and a third level; The first level includes the subject classification, title, and abstract of academic papers; The second level includes the academic paper's subject classification, title, abstract, and introduction; The third level includes the full text of academic papers, excluding citations. The metadata of the first level, the second level, and the third level is consistent, and the metadata includes the author information, publication year, and publication location information of the academic paper.

4. The academic citation prediction method according to claim 3, characterized in that, The method, based on a pre-built multi-level corpus, performs a multi-level hybrid retrieval based on the queried paper information to obtain the target retrieval results, including: The queried paper information is encoded into a query embedding vector; Based on the query embedding vector, similarity searches are performed at the first, second, and third levels of the multi-level corpus, respectively, and the first-level search results, the second-level search results, and the third-level search results are returned. The results of the first level of search, the second level of search, and the third level of search are merged using the inverse ranking method, and the fusion score of multiple candidate academic papers is calculated. The candidate academic papers are reordered based on the fusion score to obtain the target retrieval results; the target retrieval results include the titles, authors, publication years, and abstracts of the retrieved papers.

5. The academic citation prediction method according to claim 4, characterized in that, Based on a pre-trained retrieval tool, similarity searches are performed at the first, second, and third levels of the multi-level corpus according to the query embedding vector, and the retrieval results at the first, second, and third levels are returned. The retrieval system is built on a large language model and is fine-tuned by comparing and learning from positive and negative query sample pairs.

6. The academic citation prediction method according to claim 5, characterized in that, Also includes: The retriever was evaluated using recall and average reciprocal ranking. The list-level citation prediction task was evaluated using recall, normalized depreciation cumulative gain, and hit rate. The location-level citation prediction task is evaluated using location-aware citation accuracy. The generative model was evaluated using citation diversity entropy and illusion rate.

7. The academic citation prediction method according to any one of claims 1-6, characterized in that, Each sample in the list-level citation prediction sample set includes the title, abstract, and complete citation list of the queried paper sample; Each sample in the location-level citation prediction sample set includes the title, abstract, text paragraphs including citation placeholders, and the corresponding list of actual citations for the queried paper sample.

8. An academic citation prediction device, characterized in that, include: The paper information determination module is used to determine the paper information corresponding to the current citation prediction task. The current citation prediction task includes list-level citation prediction tasks and position-level citation prediction tasks, and the queried paper information includes at least the title and abstract of the queried paper. The multi-level hybrid retrieval module is used to perform multi-level hybrid retrieval based on a pre-built multi-level corpus and the queried paper information to obtain the target retrieval results; The academic citation prediction module is used to generate target citation prediction results based on the queried paper information and the target retrieval results, using a pre-trained generative model. The generative model is built based on a large language model and is obtained by training and optimizing it based on a list-level citation prediction sample set and a position-level citation prediction sample set.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the academic citation prediction method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the academic citation prediction method as described in any one of claims 1 to 7.

Images