An academic citation deviation detection method based on a large language model

Abstract

The application discloses a kind of academic citation deviation detection methods based on large language model, by parsing the academic document to be detected, identify the anchor point in the text, reference object and its corresponding reference article entry;With the center of reference anchor point, extract the context window before and after, form the context of to-be-verified assertion;Based on DOI, local literature library identifier, reference title or full-text file path, obtain the structured evidence content of cited literature, and select candidate evidence fragments from abstract, conclusion, text paragraph or page number fragment;Construct four-dimensional decoupling evaluation task, call large language model to output classification labels, confidence and evidence position of assertion support degree, causal direction consistency, timeliness effectiveness and representativeness rationality respectively;Weighted aggregation is carried out to four-dimensional results, and the overall citation deviation type and comprehensive risk confidence are obtained;According to threshold value, trigger in-document annotation, hyperlink, page number prompt, replacement suggestion and audit log generation.

Description

Technical Field

[0001] This invention relates to the interdisciplinary fields of natural language processing, intelligent document processing, and academic quality verification, specifically to an academic citation deviation detection method based on a large language model. Background Technology

[0002] Academic documents such as academic papers, dissertations, and research reports typically establish connections between specific arguments and existing literature through in-text citations. Citations not only serve a formatting function but also provide supporting evidence, facilitate knowledge tracing, and assign academic responsibility.

[0003] In the actual writing and review process, semantic deviations may exist between the arguments in the main text and the actual content of the cited literature. For example, the main text may not be supported by the original text; it may present a correlation as a causal relationship; it may use outdated conclusions that have been revised by subsequent research; or it may present minority viewpoints, regional conclusions, or fringe work as consensus in the field. Such deviations reduce the credibility and verifiability of academic texts.

[0004] Existing reference management tools typically focus on generating reference formats, updating citation numbers, managing reference entries, and managing full-text attachments, but they cannot automatically determine whether there is a semantic deviation between the main text's arguments and the content of the cited references. Some writing aids can identify citation format errors or missing references, but they still struggle to perform engineered checks on the supporting relationships within the cited content itself.

[0005] When using a large language model to make a single-hint judgment on the correctness of a citation, multiple discriminative dimensions can easily be coupled in the same reasoning process. The model may output general conclusions such as "basically supported" or "possibly inconsistent," but it cannot clearly distinguish whether the argument is not supported, the causal direction is wrong, the timeliness is insufficient, or the representativeness is inappropriate, resulting in insufficient accuracy, interpretability, and auditability of the judgment.

[0006] Furthermore, citations in academic documents often have complex structures. Multiple citations may be contained within the same brackets, the same author may correspond to multiple years, and the same author and year may have "a" or "b" suffixes. Inconsistencies may also exist in titles, author abbreviations, DOIs, and full-text attachments across different bibliographic databases. If disambiguation of citations and localization of evidence fragments cannot be completed before detection, the judgment results of the large language model are easily influenced by incorrect context or erroneous bibliographic entries.

[0007] Therefore, it is necessary to provide a citation deviation detection method that can automatically parse citation objects in academic documents, locate evidence fragments of cited literature, decompose compound citation deviation judgments into multiple auditable semantic dimensions, and feed the detection results back to the document editing process.

[0008] To address the aforementioned issues, this invention provides an academic citation deviation detection method based on a large language model, aiming to solve problems in existing technologies such as the inability of citation management tools to determine semantic compliance, the coupling of judgment dimensions in single-prompt large language models, insufficient disambiguation of citation objects, untraceable evidence page numbers, and difficulty in feeding back detection results to the document revision process. Summary of the Invention

[0009] The purpose of this invention is to provide a method for detecting academic citation deviations based on a large language model. The specific steps of the method are as follows: S1. Parse the academic document to be examined and identify the citation anchors, citation text, citation objects, and the reference entries corresponding to the citation objects in the text. S2. Centered on the citation anchor point, the academic document to be examined is subjected to sentence boundary recognition, citation anchor point location, same sentence assertion recognition, and sentence expansion before and after. At least one sentence before the citation anchor point, the sentence containing the citation anchor point, and at least one sentence after the citation anchor point are extracted. The window boundary is expanded or truncated according to whether the citation anchor point is located in a parallel clause, a transitional sentence, or a chart description, thus forming a context window for the assertion to be verified. S3. Obtain the evidence content of the cited literature based on the bibliographic identifier or bibliographic metadata of the bibliographic entries, and identify candidate evidence fragments related to the context window of the argument to be verified from the evidence content. S4. Input the context window of the argument to be verified and the candidate evidence fragments into the large language model or natural language reasoning model, construct independent evaluation tasks for the argument support, consistency of causal direction, timeliness and validity, and representativeness and reasonableness, respectively, and output the classification label, confidence, evidence location, evidence text and explanatory text corresponding to each semantic dimension, and verify the completeness of the output fields and the validity of the evidence location. S5. Map the classification labels corresponding to multiple preset semantic dimensions to risk scores, verify the validity of the confidence of each semantic dimension, determine and normalize the dimension weights according to the subject area, document type, citation purpose or user preset strategy, and perform weighted aggregation according to risk scores, confidence levels and dimension weights to obtain the overall citation deviation type and comprehensive risk confidence. S6. When the overall citation deviation type or comprehensive risk confidence level meets the preset trigger conditions, perform the corresponding prompts, annotations, hyperlinks, page numbering, replacement suggestions or audit log generation operations on the academic document to be inspected.

[0010] Furthermore, the specific steps for constructing the context window for the unverified assertion in S2 are as follows: S21. Segment the main text paragraphs, figure titles, table titles, footnotes, or endnotes of the academic document to be examined into sentences, and record the paragraph number and character offset of each sentence; S22. Determine the sentence containing the anchor point. If there are multiple reference objects in the same sentence, generate detection tasks for each reference object separately. S23. Extract the sentence containing the anchor point, the preceding N sentences, and the following M sentences as the base window, where N and M are configurable parameters; S24. When the sentence containing the anchor point contains a causal, adversative, parallel, or referential relationship, the adjacent clauses or sentences connected to that relationship will be merged into the window. S25. Delete invalid text containing only format information, table of contents information, or reference entry information to obtain the context window of the statement to be verified.

[0011] Furthermore, the process of identifying candidate evidence fragments in S3 includes segmenting the full text of the cited literature into paragraphs and mapping page numbers, calculating the semantic similarity between the context window of the argument to be verified and each paragraph, selecting paragraphs with similarity higher than a preset threshold and their adjacent paragraphs as candidate evidence fragments, and retaining the page numbers, chapter titles and original text location indexes corresponding to the candidate evidence fragments.

[0012] Furthermore, S4 includes several pre-defined semantic dimensions, including assertion support, causal direction consistency, timeliness and validity, and representativeness and reasonableness. Assertion support is used to determine whether the cited literature supports the assertion in the text. Causal direction consistency is used to determine whether the causal, influence, or control relationship in the text is consistent with the cited literature. Timeliness and validity is used to determine whether the cited conclusion has been corrected or overturned by subsequent research. Representativeness and reasonableness is used to determine whether the text inappropriately presents a minority viewpoint or marginal work as a consensus in the field.

[0013] Furthermore, S4 includes several pre-defined semantic dimensions such as assertion support, causal consistency, timeliness, and representativeness. The specific steps of the decoupling evaluation process are as follows: S41. Construct independent prompt templates or independent reasoning tasks for each semantic dimension. S42. Input the context window of the argument to be verified, candidate evidence fragments, reference metadata, and citation usage description into each independent evaluation task; S43. Output the classification label, confidence score, evidence page number, evidence text, and explanation text respectively from the large language model or the natural language inference model; S44. Perform format verification and evidence location verification on the output of each dimension; S45. When any dimension output lacks necessary fields, the evidence location cannot be located, or the confidence level is lower than the preset threshold, the referenced object is marked as awaiting manual review.

[0014] Furthermore, the specific steps of S5's weighted aggregation are as follows: S51. Map the classification labels of each semantic dimension to risk scores, where full support or no significant deviation corresponds to a low-risk score, while partial support, insufficient evidence, unstated causality, outdated conclusions, or insufficient representativeness correspond to a higher-risk score. S52. Read the confidence scores of each semantic dimension and perform normalization or threshold verification; S53. Determine the dimension weights based on the subject area, document type, citation purpose, or user-preset strategy, and make the sum of the weights of each dimension equal to 1; S54. Calculate the comprehensive risk confidence score according to R=Σ_i(w_i×r_i×p_i), where r_i is the risk score of the i-th semantic dimension, p_i is the corresponding confidence score, and w_i is the weight of the corresponding dimension. S55. Determine the overall reference deviation type based on the comprehensive risk confidence level and the classification label of the highest risk dimension.

[0015] Furthermore, the specific steps of S6 are as follows: S61. Color-coded or annotated references are used to mark suspected deviations in academic documents to be inspected. S62. Create hyperlinks for the cited objects to the reference entries and the cited evidence fragments; S63. Record the corresponding page number for the location where the same reference is cited multiple times; S64. Generate an audit log that includes citation anchors, reference entries, candidate evidence fragments, four-dimensional assessment results, comprehensive risk confidence level, processing actions, and manual review status.

[0016] Furthermore, the overall citation deviation types in S5 include no obvious deviation, weakly supporting citation, irrelevant evidence, deviation from causal direction, outdated citation, representativeness deviation, and requiring manual review; when the overall risk confidence exceeds the first threshold, a suspected deviation prompt is output; when the overall risk confidence exceeds the second threshold or any key dimension is judged to be high risk, the document annotation and manual review process is triggered.

[0017] The present invention has the following advantages: by decomposing the reference deviation detection into multiple relatively orthogonal semantic dimensions, the present invention avoids coupling misjudgment between multiple discrimination targets in a single prompt large language model, thereby improving the stability and interpretability of the detection results.

[0018] This invention reduces the risk of misjudgment due to incorrect citation matching by performing citation object identification, disambiguation, and evidence fragment localization before detection, enabling model evaluation to be based on clear citation objects and traceable evidence content.

[0019] This invention enables citation deviation detection results to be manually reviewed and automatically consumed by document processing programs through structured output, evidence page number retention, and audit log generation.

[0020] This invention improves the efficiency of academic document quality verification by writing the detection results back to the academic document to be inspected, thereby achieving closed-loop processing from citation identification, evidence verification, deviation judgment to document annotation and remedial suggestions. Attached Figure Description

[0021] Figure 1 This is a flowchart of the present invention; Figure 2 This is a schematic diagram of the process for disambiguating references and locating evidence fragments in this invention; Figure 3 This is a schematic diagram of the four-dimensional decoupling evaluation and document feedback closed loop of the present invention. Detailed Implementation

[0022] This invention provides a method for detecting academic citation deviations based on a large language model.

[0023] Example 1 (refer to) Figure 1 The method provided in this embodiment is deployed on a computing system equipped with a document parsing environment, a literature database access environment, a full-text parsing environment, and a large language model calling environment. The literature database access environment can access the local Zotero RDF literature database, EndNote exported files, BibTeX files, DOI interfaces, full-text PDF catalogs, or other literature metadata sources.

[0024] S1. First, identify citation anchors from the academic document to be inspected: Citation anchors can be located in body paragraphs, figure titles, table titles, footnotes, or endnotes. For author-year citations, the system identifies the author, year, and necessary suffixes; for sequentially numbered citations, the system identifies the number and maps it to the reference list; for multiple citations within the same brackets, the system splits them into multiple citation objects and establishes separate detection tasks.

[0025] S2. Reference Object Identification and Disambiguation: Refer to Figure 2 The document parsing module reads text paragraphs, field codes, annotations, hyperlinks, and reference chapters from the DOCX file. The system locates cited text based on regular expressions and a natural language recognition model, and generates initial citation records. Each initial citation record contains at least the document location, paragraph number, cited text, contextual text, candidate author, candidate year, and citation type.

[0026] For multiple citations by the same author in the same year of the type “Wu et al. (2021, 2022)”, the system splits them into two citation objects: Wu-2021 and Wu-2022. For multiple citations by the same author in the same year of the type “Shen et al., 2013a,b”, the system combines the suffix, title, abstract keywords and contextual semantics of the reference entries to perform disambiguation.

[0027] When multiple reference entries match in both author and year, the system calculates the similarity between the citation context and the candidate reference's title, abstract, and keywords. It then generates a ranking based on similarity, DOI consistency, the existence of full-text attachments, and user confirmation records. If the highest-ranking candidate is still below a preset reliability threshold, the citation is marked for manual review to prevent erroneous references from being submitted for subsequent semantic evaluation.

[0028] S3. Context Window and Evidence Fragment Construction: Refer to Figure 2 For each disambiguated reference, the system extracts the first N sentences and the last M sentences centered on the reference anchor point to form a context window W for the argument to be verified. N and M are configurable parameters. In one embodiment, N=1 and M=1. In a summary paragraph, the window range can be increased to cover the complete argument.

[0029] The system retrieves the cited literature evidence content C based on the reference entries. If a full-text PDF exists, it performs text extraction, OCR recognition, paragraph segmentation, and page number mapping on the full text; if a full-text PDF does not exist, it prioritizes using the abstract, keywords, conclusions, or notes fields from the literature database. For full-text documents with page number mapping, the system saves the start and end page numbers of each evidence segment.

[0030] The system calculates the semantic similarity between the context window W and each evidence segment in the cited literature, and selects one or more segments with the highest similarity as candidate evidence segments E. To avoid misjudgment due to overly narrow evidence segments, the system can include adjacent segments before and after the most similar segment in the candidate evidence set.

[0031] S4. Four-Dimensional Decoupling Assessment: Refer to Figure 3 The system constructs four independent evaluation tasks for each cited object. The input of each task includes the context window W of the argument to be verified, the candidate evidence fragment E, the reference metadata, and the necessary description of the purpose of the citation, but the output targets are different.

[0032] The first dimension is the assertion support level, used to determine whether E supports the core assertion in W. The system requires the model to output "fully supported," "partially supported," "not supported," or "irrelevant," along with the corresponding evidence sentences.

[0033] The second dimension is causal direction consistency, used to determine whether the relationships expressed in W (such as cause, control, promotion, inhibition, constraint, and indication) are consistent with the relationship direction in E. If E only describes correlation without declaring causality, the system outputs the original text as "no causality declared".

[0034] The third dimension is timeliness and validity, used to determine whether the conclusions in E can still be used as the basis for current arguments. The system can call subsequent literature collections, review literature collections, or user-specified updated literature databases to determine whether the conclusions have been revised or refuted; if there is a lack of subsequent evidence, the system will output "insufficient evidence".

[0035] The fourth dimension is representativeness and reasonableness, used to determine whether W inappropriately presents case studies, regional conclusions, minority viewpoints, or fringe work from E as consensus within the field. This dimension can be assessed in conjunction with the type of literature, review level, citation frequency, scope of research subjects, and qualifiers in the evidence.

[0036] Each dimension's output uses a structured format, including the dimension name, category label, confidence score, evidence page number, evidence text, natural language explanation, and whether manual review is required. The system performs format validation on the structured output; if a field is missing or the evidence page number cannot be located, a re-evaluation or manual review is triggered.

[0037] S5. Weighted Aggregation and Deviation Type Determination: Refer to... Figure 3 The system maps the four-dimensional classification labels to risk scores r_i, and denotes the corresponding confidence scores as p_i and the dimension weights as w_i. The overall risk confidence score R can be calculated using the following formula: R = Σ_i (w_i × r_i × p_i) Where i represents the semantic dimension number, Σ_i w_i = 1, for disciplines that emphasize the support of facts, the weight of the support of the argument can be increased; for review writing, the weight of representativeness, rationality and timeliness can be increased; for paragraphs arguing for causal mechanisms, the weight of consistency of causal direction can be increased.

[0038] The system determines the overall citation deviation type based on R and the labels of each dimension. The overall citation deviation type includes, but is not limited to, no obvious deviation, weak supporting citation, irrelevant evidence, deviation from causal direction, outdated citation, representativeness deviation, and citation requiring manual review.

[0039] S6. Intra-document closed-loop feedback and audit log: Refer to Figure 3 When the system determines that there is a risk of deviation in the citation, corresponding processing actions can be performed on the academic document to be inspected. For example, citations that cannot be matched with references are highlighted in red, weakly supported citations are highlighted in yellow, superscripts are added to page numbers of duplicate citations that have been confirmed to be supported but require additional page numbers, and hyperlinks are created to the evidence fragments pointing to the local PDF or literature database entries.

[0040] The system can also generate replacement suggestions. When a citation is deemed unsupported or insufficiently representative, the system can retrieve semantically more relevant candidate documents from the local bibliography or a user-specified bibliography set, and add these candidate documents to the suggestion list pending manual confirmation. Replacement suggestions must not be automatically written into the main text unless confirmed by the user.

[0041] Audit logs are used to ensure that the detection process is verifiable. Each log entry must include at least the document file name, citation location, original citation text, parsed citation object, matched reference entry, evidence fragment page number, four-dimensional output, comprehensive risk confidence level, system processing action, manual confirmation status, and timestamp.

[0042] Example 2

[0043] In a thesis processing scenario, the system reads a DOCX file containing text citations and reference chapters, and reads the user-provided local RDF library and PDF full-text catalog. The system identifies all citations in the text, establishes the correspondence between citation anchors and reference entries, and extracts the context window of each citation location when the same reference appears multiple times in the text, and locates the corresponding page number based on the PDF full text.

[0044] When a paragraph states "a certain document shows that mechanism A leads to phenomenon B", but the cited document only describes the correlation between A and B and does not declare a causal relationship, the assertion support dimension may output partial support, and the causal direction consistency dimension may output that the original text does not declare a causal relationship. After weighted aggregation, the system marks the citation as insufficient causal evidence and generates a note in the document, prompting the author to rewrite the assertion or supplement with more direct mechanism literature.

[0045] When a paragraph presents the conclusions of a case study as a consensus in the field, the representativeness dimension outputs minority viewpoints or marginal work. The system may suggest changing the statement from "generally believed" to "observed in a certain study area," or suggest supplementing it with review literature as support.

[0046] Although specific embodiments of the present invention have been described in detail with reference to the accompanying drawings, this should not be construed as limiting the scope of protection of the present invention. Various modifications and variations that can be made by those skilled in the art without inventive effort within the scope described in the claims are still within the scope of protection of the present invention.

Claims

1. A method for detecting academic citation deviation based on a large language model, characterized in that: The specific steps of the method are as follows: S1. Parse the academic document to be examined and identify the citation anchors, citation text, citation objects, and the reference entries corresponding to the citation objects in the text. S2. Centered on the citation anchor point, the academic document to be examined is subjected to sentence boundary recognition, citation anchor point location, same sentence assertion recognition, and sentence expansion before and after. At least one sentence before the citation anchor point, the sentence containing the citation anchor point, and at least one sentence after the citation anchor point are extracted. The window boundary is expanded or truncated according to whether the citation anchor point is located in a parallel clause, a transitional sentence, or a chart description, thus forming a context window for the assertion to be verified. S3. Obtain the evidence content of the cited literature based on the bibliographic identifier or bibliographic metadata of the bibliographic entries, and identify candidate evidence fragments related to the context window of the argument to be verified from the evidence content. S4. Input the context window of the argument to be verified and the candidate evidence fragments into the large language model or natural language reasoning model, construct independent evaluation tasks for the argument support, consistency of causal direction, timeliness and validity, and representativeness and reasonableness, respectively, and output the classification label, confidence, evidence location, evidence text and explanatory text corresponding to each semantic dimension, and verify the completeness of the output fields and the validity of the evidence location. S5. Map the classification labels corresponding to multiple preset semantic dimensions to risk scores, verify the validity of the confidence of each semantic dimension, determine and normalize the dimension weights according to the subject area, document type, citation purpose or user preset strategy, and perform weighted aggregation according to risk scores, confidence levels and dimension weights to obtain the overall citation deviation type and comprehensive risk confidence. S6. When the overall citation deviation type or comprehensive risk confidence level meets the preset trigger conditions, perform the corresponding prompts, annotations, hyperlinks, page numbering, replacement suggestions or audit log generation operations on the academic document to be inspected.

2. The academic citation deviation detection method based on a large language model as described in claim 1, characterized in that: The specific steps for constructing the context window for the assertion to be verified in S2 are as follows: S21. Segment the main text paragraphs, figure titles, table titles, footnotes, or endnotes of the academic document to be examined into sentences, and record the paragraph number and character offset of each sentence; S22. Determine the sentence containing the anchor point. If there are multiple reference objects in the same sentence, generate detection tasks for each reference object separately. S23. Extract the sentence containing the anchor point, the preceding N sentences, and the following M sentences as the base window, where N and M are configurable parameters; S24. When the sentence containing the anchor point contains a causal, adversative, parallel, or referential relationship, the adjacent clauses or sentences connected to that relationship will be merged into the window. S25. Delete invalid text containing only format information, table of contents information, or reference entry information to obtain the context window of the statement to be verified.

3. The academic citation deviation detection method based on a large language model as described in claim 1, characterized in that: The evidence content in S3 includes the title, abstract, keywords, conclusion, full text paragraphs, table descriptions, figure captions, and page number information of the cited literature. The evidence content is obtained through at least one of the following methods: DOI interface, local RDF literature database, local PDF full text, OCR text, file exported from literature management software, and manual supplementation of the full text path.

4. The academic citation deviation detection method based on a large language model as described in claim 1, characterized in that: The process of determining candidate evidence fragments in S3 includes segmenting the full text of the cited literature into paragraphs and mapping page numbers, calculating the semantic similarity between the context window of the argument to be verified and each paragraph, selecting paragraphs with similarity higher than a preset threshold and their adjacent paragraphs as candidate evidence fragments, and retaining the page number, chapter title and original text position index corresponding to the candidate evidence fragments.

5. The academic citation deviation detection method based on a large language model as described in claim 1, characterized in that: The S4 framework includes multiple preset semantic dimensions such as assertion support, causal consistency, timeliness, and representativeness; the specific steps of the decoupling evaluation process are as follows: S41. Construct independent prompt templates or independent reasoning tasks for each semantic dimension. S42. Input the context window of the argument to be verified, candidate evidence fragments, reference metadata, and citation usage description into each independent evaluation task; S43. Output the classification label, confidence score, evidence page number, evidence text, and explanation text respectively from the large language model or the natural language inference model; S44. Perform format verification and evidence location verification on the output of each dimension; S45. When any dimension output lacks necessary fields, the evidence location cannot be located, or the confidence level is lower than the preset threshold, the referenced object is marked as awaiting manual review.

6. The academic citation deviation detection method based on a large language model as described in claim 1, characterized in that: The specific steps of the weighted aggregation in S5 are as follows: S51. Map the classification labels of each semantic dimension to risk scores, where full support or no significant deviation corresponds to a low-risk score, while partial support, insufficient evidence, unstated causality, outdated conclusions, or insufficient representativeness correspond to a higher-risk score. S52. Read the confidence scores of each semantic dimension and perform normalization or threshold verification. S53. Determine the dimension weights based on the subject area, document type, citation purpose, or user preset strategy, and make the sum of the weights of each dimension equal to 1. S54. Calculate the comprehensive risk confidence score according to R=Σ_i(w_i×r_i×p_i), where r_i is the risk score of the i-th semantic dimension, p_i is the corresponding confidence score, and w_i is the weight of the corresponding dimension. S55. Determine the overall reference deviation type based on the comprehensive risk confidence level and the classification label of the highest risk dimension.

7. The academic citation deviation detection method based on a large language model as described in claim 1, characterized in that: The specific steps of S6 are as follows: S61. Color-coded or annotated references are used to mark suspected deviations in academic documents to be inspected. S62. Create hyperlinks for the cited objects to the reference entries and the cited evidence fragments; S63. Record the corresponding page number for the location where the same reference is cited multiple times; S64. Generate an audit log that includes citation anchors, reference entries, candidate evidence fragments, four-dimensional assessment results, comprehensive risk confidence level, processing actions, and manual review status.

8. The academic citation deviation detection method based on a large language model as described in claim 1, characterized in that: The overall citation deviation types in S5 include no obvious deviation, weakly supporting citation, irrelevant evidence, deviation in causal direction, outdated citation, representativeness deviation, and citation requiring manual review. When the overall risk confidence exceeds the first threshold, a suspected deviation prompt is output. When the overall risk confidence exceeds the second threshold or any key dimension is judged to be high risk, the document annotation and manual review process is triggered.

Images