A kind of reference authentication method, device, equipment, medium and computer program product

Abstract

This invention discloses a citation verification method, apparatus, device, medium, and computer program product, comprising: acquiring original input; extracting the claim text to be verified and a candidate citation list associated with the claim text from the original input to form structured citation information and opinion citation pairs; performing existence verification on each candidate citation in the candidate citation list based on the structured citation information to determine whether the target document corresponding to the candidate citation actually exists and is accessible, generating an existence verification result, which at least includes a reason code and / or a hit method; and performing semantic alignment and support determination on the claim text and evidence fragments in the target document based on the opinion citation pairs to generate an opinion verification conclusion, which at least includes suggested actions and / or evidence location information. These technical features automate the entire citation verification process, improving the accuracy, traceability, and engineering reusability of citation verification.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a citation verification method, apparatus, device, medium, and computer program product. Background Technology

[0002] With the widespread application of large-scale models in scientific research retrieval, paper reading, review writing, and knowledge-based Q&A, "answering with citations" has become the mainstream interaction method. Citations must meet four requirements: findable, locatable, aligned, and auditable.

[0003] Current citation verification schemes mainly fall into four categories: retrieval without verification (only obtaining the document title or link without verifying its authenticity); existence verification only (verifying the existence of the document without assessing its content's support); manual verification (requiring manual reading and locating of evidence, which is costly and difficult to scale); and weak alignment strategies (relying on keywords or simple similarity comparisons, which are prone to inaccuracies in complex semantic scenarios). All of these schemes fail to meet the practical requirements for citation reliability in current large-scale scientific research applications. Summary of the Invention

[0004] This invention provides a method, apparatus, device, medium, and computer program product for citation verification, which automates the entire citation verification process and improves the accuracy, traceability, and engineering reusability of citation verification.

[0005] In a first aspect, embodiments of the present invention provide a method for verifying references, including: Obtain the original input, extract the claim text to be verified and the candidate citation list associated with the claim text from the original input, and form structured citation information and opinion citation pairs; Based on the structured citation information, an existence check is performed on each candidate citation in the candidate citation list to determine whether the target document corresponding to the candidate citation actually exists and can be obtained, and an existence check result is generated. The existence check result includes at least a reason code and / or a hit method. Based on the aforementioned viewpoint citation pairs, semantic alignment and support determination are performed on the assertion text and evidence fragments in the target document to generate viewpoint verification conclusions. The viewpoint verification conclusions include at least suggested actions and / or evidence location information.

[0006] Secondly, embodiments of the present invention provide a reference verification device, comprising: The input preprocessing module is used to acquire the raw input, extract the claim text to be verified and the candidate citation list associated with the claim text from the raw input, and form structured citation information and opinion citation pairs; An existence verification module is used to perform an existence verification on each candidate citation in the candidate citation list based on the structured citation information, so as to determine whether the target document corresponding to the candidate citation actually exists and can be obtained, and generate an existence verification result. The existence verification result includes at least a reason code and / or a hit method. The opinion verification module is used to perform semantic alignment and support determination on the claim text and the evidence fragments in the target document based on the opinion citation pair, and generate an opinion verification conclusion. The opinion verification conclusion includes at least the suggested action and / or evidence location information.

[0007] Thirdly, embodiments of the present invention provide an electronic device, including: At least one processor; and A memory that is communicatively connected to at least one processor; wherein, The memory stores a computer program that can be executed by at least one processor, such that the at least one processor is able to perform a reference verification method provided in the first aspect embodiment described above.

[0008] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing computer instructions that are used to cause a processor to execute and implement a reference verification method provided in the first aspect of the embodiments described above.

[0009] Fifthly, embodiments of the present invention provide a computer program product, the computer program product including a computer program, which, when executed by a processor, implements a reference verification method provided in the first aspect of the embodiments described above.

[0010] The technical solution of this invention involves acquiring original input, extracting the claim text to be verified and a list of candidate citations associated with the claim text from the original input, forming structured citation information and opinion citation pairs; based on the structured citation information, performing an existence check on each candidate citation in the candidate citation list to determine whether the target document corresponding to the candidate citation truly exists and is accessible, generating an existence check result, which at least includes a reason code and / or a hit method; based on the opinion citation pairs, performing semantic alignment and support determination on the claim text and evidence fragments in the target document, generating an opinion verification conclusion, which at least includes suggested actions and / or evidence location information. These technical features, by extracting the claim text and candidate citations to form structured citation information and opinion citation pairs, first performing existence checks on the citations to confirm the target document is truly accessible, and then performing semantic alignment and support determination on the claim and evidence fragments, solve the problems of existing technologies that only retrieve without verification, only check existence, have high manual costs, and are prone to inaccuracies due to weak alignment. Compared to traditional solutions, this invention can automatically complete citation authenticity verification, evidence location, and semantic support judgment, forming standardized and auditable verification conclusions, significantly reducing the cost of manual verification and enabling large-scale implementation; at the same time, it outputs structural specifications, which are easy to integrate into scientific research retrieval, writing, question answering and other systems, significantly improving the reliability, traceability and credibility of citations generated by large models.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of a reference verification method provided in an embodiment of the present invention; Figure 2 This is a signaling interaction diagram of a reference verification method provided in an embodiment of the present invention; Figure 3 This is a flowchart of input preprocessing involved in a reference verification method provided in an embodiment of the present invention; Figure 4 This is a flowchart involving existence verification in a reference verification method provided in an embodiment of the present invention; Figure 5 This is a flowchart of a reference verification method provided in an embodiment of the present invention, involving the verification of viewpoints. Figure 6 This is a schematic diagram of the structure of a reference verification device provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] In applications such as generative question answering and scientific writing, model outputs often contain unverifiable, miscited, or inconsistent paper citations, resulting in low citation credibility and difficulty in auditing and reproducing the findings. Existing systems generally lack a complete, engineerable technical solution that integrates "citation existence verification + evidence fragment location + claim-evidence support determination + structured result output."

[0017] Currently, existing solutions typically fall into the following categories: 1. Retrieval without verification: Several paper titles / links are obtained through retrieval, and then the model generates citations freely, lacking verification of each one; 2. Existence verification only: Verify the existence of DOI / title, but do not determine whether it supports the claim; 3. Manual verification: Rely on manual reading of papers to locate evidence, which is costly and not scalable; 4. Weak alignment strategy: Perform keyword matching or simple similarity comparison, which is prone to inaccuracies in scenarios such as paraphrasing, cross-sentence reasoning, and negation / conditional semantics.

[0018] However, simply providing a link / title cannot prove that the original paper actually supports the claim, nor can it prove its support; the lack of paragraph / page number / location makes quick verification impossible, and the evidence is not traceable; the absence of a unified input / output protocol and result structure makes it difficult to reuse by modules such as search / writing / in-depth research, resulting in a missing engineering loop; and the lack of hierarchical judgment and confidence assessment makes it impossible to provide risk warnings and downgrades on the product side, making the quality uncontrollable.

[0019] Based on this, the present invention provides a method, device, electronic device, storage medium, and computer program product for verifying paper citations based on claim-evidence alignment, in order to solve the problem of how to automatically complete the verification of the authenticity and support of citations given a claim text and a set of candidate documents, and output a traceable chain of evidence (located to page number / paragraph / sentence) and verification conclusion, so as to reduce the risk of illusory citations and improve the credibility of scientific research content.

[0020] In one embodiment, Figure 1 This is a flowchart of a citation verification method provided by an embodiment of the present invention. This embodiment can be applied to situations where the existence verification and opinion verification of literature citations are performed during question-and-answer processes. The method can be executed by a citation verification device, which can be implemented in hardware and / or software.

[0021] like Figure 1 As shown, the method includes: S101. Obtain the original input, extract the claim text to be verified and the candidate citation list associated with the claim text from the original input, and form structured citation information and opinion citation pairs.

[0022] In this embodiment, the original input can be understood as the verifiable information uploaded by the user or transmitted from the business side, including the verifiable text, Portable Document Format (PDF) files, Uniform Resource Locator (URL) links, and related literature resources. The claim text can be understood as factual assertions, comparative conclusions, or numerical arguments extracted from the original input that require documentary evidence, i.e., "opinions," such as the text content of the verifiable text, PDF files, or URL links. The candidate citation list can be understood as a collection of original citation entries directly extracted from the original input that have not yet undergone existence verification. The structured citation information can be understood as uniformly formatted data formed after format normalization and field standardization of the candidate citation list, including citation fields such as title, author, publication year, Digital Object Identifier (DOI), ArXiv ID, and publication journal / conference, organized in a unified key-value pair format for easy direct access by subsequent modules. A viewpoint citation pair can be understood as the smallest verification unit formed by binding a viewpoint in the text of the claim to be verified with one or more corresponding candidate citations.

[0023] Specifically, after obtaining the original input containing the content to be verified and relevant literature, it is first determined whether the input is academic text. If the input is academic text, the corresponding extraction strategy is adopted according to the type of input carrier (text, PDF file, URL link) to extract candidate citation information and complete the format normalization processing of author, year, punctuation, abbreviation, etc. to form structured citation information. At the same time, the input text is divided into sentences and paragraphs, factual assertions and numerical conclusions are identified, the text of the claim to be verified is extracted, and the claim text is bound with the associated candidate citations to form viewpoint citation pairs.

[0024] S102. Based on the structured citation information, perform an existence check on each candidate citation in the candidate citation list to determine whether the target document corresponding to the candidate citation actually exists and can be obtained, and generate an existence check result. The existence check result shall at least include a reason code and / or a hit method.

[0025] In this embodiment, existence verification can be understood as the process of retrieving and verifying whether the target document corresponding to the candidate citation truly exists and can be obtained normally, based on structured citations. The existence verification result includes at least a reason code and / or a hit method; in specific implementations, the existence verification result may also include fields such as verification conclusion, document metadata, and confidence level.

[0026] Specifically, following the strategy of prioritizing strong identifiers and supplementing with weak identifiers and combined identifiers, each document in the candidate citation list is searched sequentially to obtain a candidate document set, which is then filtered and disambiguated to determine the target document. Next, the metadata of the target document is obtained through the document retrieval interface, and the consistency between the target document and the corresponding candidate citations in the candidate citation list is verified to determine whether the target document actually exists and is accessible, and finally, an existence verification result is generated.

[0027] S103. Based on the viewpoint citation pair, perform semantic alignment and support determination on the assertion text and evidence fragments in the target document to generate a viewpoint verification conclusion. The viewpoint verification conclusion shall at least include suggested actions and / or evidence location information.

[0028] In this embodiment, semantic alignment and support determination can be understood as the process of determining whether the claim text and the evidence fragments in the target document are semantically consistent and whether they can form a supporting relationship. The opinion verification conclusion includes at least the suggested action and / or evidence location information; in specific implementations, the opinion verification conclusion may also include fields such as verification result, corresponding evidence fragments, and reason code, and the verification result includes at least support, non-support, and doubt.

[0029] Specifically, taking opinion citation pairs as units, lightweight evidence assessment is performed based on summaries or structured fields, outputting an opinion verification conclusion that includes the verification result, corresponding evidence fragments, and suggested actions. When the lightweight evidence assessment is insufficient, the entire text is further parsed to retrieve paragraph-level or sentence-level evidence fragments. The claim text, context, and evidence fragments are semantically aligned to complete the support assessment, ultimately outputting an opinion verification conclusion that includes the verification result, corresponding evidence fragments, evidence location information, reason codes, and suggested actions.

[0030] Understandably, after completing existence verification or opinion verification, the corresponding existence verification results or opinion verification conclusions can be stored in the database for retrospective auditing.

[0031] This invention provides a citation verification method, comprising: acquiring original input; extracting the claim text to be verified and a list of candidate citations associated with the claim text from the original input to form structured citation information and opinion citation pairs; performing existence verification on each candidate citation in the candidate citation list based on the structured citation information to determine whether the target document corresponding to the candidate citation truly exists and is accessible, generating an existence verification result, which at least includes a reason code and / or a hit method; and performing semantic alignment and support determination on the claim text and evidence fragments in the target document based on the opinion citation pairs to generate an opinion verification conclusion, which at least includes suggested actions and / or evidence location information. This technical solution, by extracting the claim text and candidate citations to form structured citation information and opinion citation pairs, first performing existence verification on the citations to confirm that the target document is truly accessible, and then performing semantic alignment and support determination on the claim and evidence fragments, solves the problems of existing technologies that only retrieve without verification, only verify existence, have high manual costs, and are prone to inaccuracies due to weak alignment. Compared to traditional solutions, this invention can automatically verify the authenticity of citations, locate evidence, and determine semantic support, forming standardized and auditable verification conclusions. This significantly reduces the cost of manual verification and allows for scalable deployment. Simultaneously, it outputs a standardized structure, facilitating integration into scientific research retrieval, writing, and question-answering systems, significantly improving the reliability, traceability, and credibility of citations generated by large models. It achieves existence and accessibility verification of citations; precise location and extraction of evidence fragments; hierarchical determination of support / non-support / doubt between claims and evidence; and a structured output of "evidence chain + conclusion + confidence level + risk reasons" for display and audit reuse in upper-level products.

[0032] As a first optional embodiment of this example, Figure 2 This is a signaling interaction diagram of a reference verification method provided in an embodiment of the present invention. Figure 3 This is a flowchart of input preprocessing involved in a reference verification method provided in an embodiment of the present invention, such as... Figure 2 and Figure 3 As shown, the original input is obtained, and the claim text to be verified and the candidate citation list associated with the claim text are extracted from the original input to form structured citation information and opinion citation pairs, including: S1011. Determine whether the original input is academic text. If not, output an inapplicable result.

[0033] In this embodiment, the original input can be understood as the data source to be verified provided by the business side, including user-uploaded PDF files, user-pasted text paragraphs, cited answers returned by generative question-and-answer systems, and accessible URL links. Academic text can be understood as research literature, reviews, paper paragraphs, or Q&A content that has a standardized citation format and contains references, footnotes, or factual arguments. The criteria for judging academic text may include whether typical citation markers (such as author years in parentheses, numerical superscripts, DOI strings), reference areas, figures, formulas, and other academic features appear in the text. The "inapplicable" result can be understood as information used to inform the caller that the current input does not need to be verified, thereby avoiding invalid calculations.

[0034] Specifically, the system first performs academic content recognition on the original input. After receiving the business input, the input preprocessing module performs academic text discrimination. If the input is purely colloquial dialogue or plain text without any citation features, the system directly returns an inapplicable / no-verification result and terminates subsequent verification. If it is determined to be academic text, it enters branch processing, selecting different extraction strategies based on the carrier type (PDF, plain text, URL). This ensures that the system only consumes computing resources on academic content that truly needs verification, improving engineering efficiency.

[0035] S1012. If so, then execute the corresponding extraction strategy according to the carrier type of the original input to obtain the claim text to be verified and the candidate citation list associated with the claim text, forming structured citation information, wherein the carrier type includes file, link or text.

[0036] In this embodiment, the carrier types include files (such as PDFs), links (URLs), and text (paragraphs pasted by the user or generated answers). Different extraction strategies are adopted for different carriers. For text paths, a large language model or rule-based methods are used to extract citation information, including title, author, year, DOI, arXiv ID, journal / conference name, etc., and a structured citation list is output. For PDF paths and URL links, text extraction is performed first, followed by text parsing of the extracted text. Citation information is then extracted using rules and a large model, and a structured citation list is output. Structured citation information refers to standardized data formed by normalizing the format and regularizing the fields of the extracted original citations. Each citation is organized into a unified key-value pair format (e.g., cite_id, title, authors, year, doi, venue, etc.), containing complete citation fields such as title, author, year, DOI, arXiv ID, journal / conference, etc., and is directly provided to the existence verification module. The claim text to be verified refers to factual statements identified from the input that need to be supported by evidence. These statements may originate from paragraphs directly entered by the user or assertions in the model-generated responses.

[0037] Specifically, once the text is identified as academic, the process proceeds according to the carrier type. If the carrier is plain text, a Large Language Model (LLM) is invoked to segment the input text into sentences or paragraphs, filtering out statements containing factual assertions, comparative relationships, or numerical conclusions to generate a list of claim texts. Simultaneously, citation locations are identified, a candidate citation list is extracted, and standardized into structured citation information. If the carrier is a URL link, the text is extracted first, and then the claim text list and candidate citation list are extracted separately according to text logic, with the candidate citation list standardized into structured citation information. If the carrier is a PDF file, the PDF undergoes text parsing and citation section location (such as reference areas, footnotes, in-text citation numbers, etc.). A rule-based + model approach is used to extract and structure citations. For Chinese citations or citations with non-standard formats, format normalization (author name, year, punctuation, abbreviations, etc.) is performed first to improve the success rate of subsequent searches. The claim text list and candidate citation list are extracted, and the candidate citation list is standardized into structured citation information. Ultimately, it is argued that textual and structured citation information are two independent standardized outputs, providing input for subsequent existence verification and opinion verification modules.

[0038] S1013. When the opinion extraction conditions are met, extract the opinion citation pairs between the claim text and the citations in the candidate citation list.

[0039] In this embodiment, the opinion extraction condition can be understood as the condition used to trigger opinion verification. For example, the input contains a factual claim that can be supported by evidence, and this claim is associated with a certain citation marker (such as the document number, author year, superscript number, etc. in parentheses) in the original text. An opinion citation pair is a tuple formed by binding the claim text with its corresponding citation entry. Each tuple contains a claim text, a candidate citation (or multiple citations) associated with the claim, and the context of the claim text (preceding and following sentences, paragraph topic). The context is used to accurately understand complex semantics such as negation, condition, and transition in subsequent support determination. If the claim text is not associated with any citation marker, the citation field in its citation pair can be left empty, and the claim can still be verified (indicating a missing citation).

[0040] Specifically, after completing citation extraction and claim extraction, the stage of claim alignment preparation begins. First, it is determined whether claim extraction is necessary. If the claim extraction conditions are not met, no further pairing is required; if the claim extraction conditions are met, claim extraction is necessary. At this point, on the one hand, the original text is segmented and the large language model is instructed to extract the claim text associated with each citation identifier (such as [1], [2-3], (a author, 2020) etc.) and output the pairing candidates in the form of (claim text, citation identifier). On the other hand, the acquired structured citation information is compressed and preprocessed. Each citation retains only the number, the first three authors, the year, and the internal index field. Other information (such as the full title, journal name, abstract, etc.) is temporarily removed to reduce the context length, avoid performance degradation of the large model when processing long sequences, and prevent redundant information from interfering with the accuracy of pairing. Then, the citation identifier in each pairing candidate is matched with the compressed and preprocessed citation information. The specific structured citation entry is located by index or number, thereby generating the final claim pair. For claims without any citation markers in the text (such as freely generated arguments by the model), attempts can be made to infer potentially cited literature through semantic similarity or co-occurrence relationships, but these are usually marked as "missing citations," forming empty citation pairs containing only the claim. Each opinion citation pair also includes the context of the claim (such as the preceding and following sentences, and the topic keywords of the paragraph) to support complex reasoning such as negation and conditional semantics in subsequent support determination. Finally, all opinion citation pairs are passed as structured output to the opinion verification module for semantic alignment and support determination by either the fast path (summary-based LLM classification) or the deep path (full-text search + alignment model). This ensures that each claim is tightly coupled with its evidence source, avoiding mismatches between evidence and claims, while significantly improving the efficiency and stability of LLM pair generation through citation information compression.

[0041] As a second optional embodiment of this example, Figure 4 This is a flowchart of an existence verification method provided in an embodiment of the present invention, such as... Figure 2 and Figure 4 As shown, based on the structured citation information, an existence check is performed on each candidate citation in the candidate citation list, including: S1021. Search each candidate citation according to the priority of the citation identifier to determine the candidate document set corresponding to the candidate citation.

[0042] The citation identifiers include strong identifiers, weak identifiers, and combined identifiers. Strong identifiers include at least a location identifier, weak identifiers include a title, and combined identifiers include at least a combination of title and author, a combination of author and year, and a combination of author and publication location.

[0043] In this embodiment, citation identifiers can be understood as key information used to locate a specific document in a literature database, and are divided into strong identifiers, weak identifiers, and combined identifiers according to their accuracy and uniqueness. Strong identifiers include at least a location identifier, such as a Digital Object Identifier (DOI) or an arXiv identifier, which can uniquely identify a document. Weak identifiers include the title, which may correspond to multiple documents of different versions or authors. Combined identifiers include at least a combination of title and author (title+author), author and year (author+year), and author and publication venue (such as the name of an academic conference or journal) (author+venue), which combine multiple fields to improve retrieval accuracy. Candidate citations can be understood as original citation entries directly extracted from the original input that have not yet undergone existence verification. Each candidate citation retains its original expression form in the original text, such as numbered citations (e.g., [1], [2-3]), author and year citations (e.g., author a, 2020), superscript numbers, or footnotes. Candidate citations can include location information, such as page number, line number, or paragraph number, facilitating retrieval of the original text. Candidate citations differ from structured citation information, which is a fielded record (including title, author, year, DOI, etc.) generated after parsing and normalizing candidate citations. Candidate citations, on the other hand, are unverified and unnormalized raw forms. A candidate document set can be understood as the collection of all potentially matching document records returned from a bibliographic database or retrieval interface based on a specific citation identifier.

[0044] Specifically, before the search, the structured citation information is checked for entity integrity. If key fields (such as DOI, title, and author) are missing, attempts are made to extract or infer them from the original candidate citations. If the key fields are not missing, each candidate citation is searched according to the priority of the citation identifiers in the key fields, trying them in the order of strong identifiers, combined identifiers, and weak identifiers. If the input contains a DOI or arXiv ID, the strong identifier is used first for precise retrieval, directly locating the candidate document set. If the strong identifier does not exist or the search is unsuccessful, combined identifiers (such as title + author, author + year) are used for retrieval in sequence. Finally, the weak identifier (title only) is used for fuzzy retrieval. After parsing, the results returned by the search are merged and deduplicated from the hit results of different identifier strategies. For each candidate document, a weighted score is calculated based on the weight of its hit identifier (e.g., 3 points for a strong identifier hit, 2 points for a combined identifier hit, and 1 point for a weak identifier hit), or the number of hits is directly used as the initial confidence value. Sort the documents in descending order of confidence level and select the top k (k is configurable, default is 3) as the candidate document set.

[0045] S1022. When the candidate document set is uniquely matched, the uniquely matched candidate document is determined as the target document.

[0046] In this embodiment, candidate documents can be understood as potentially matching document records returned from a document database or retrieval interface based on a certain citation identifier. Target documents can be understood as the actual document records identified as corresponding to the candidate citations after retrieval, filtering, and / or consistency verification.

[0047] Specifically, for the candidate document set, the confidence level (or hit count) of the first-ranked (Top1) and the second-ranked (Top2) documents is compared first. If the difference in hit count between Top1 and Top2 is greater than 1 (i.e., the difference is significant), Top1 is directly selected as the target document and considered a unique hit. If the difference is not greater than 1, all candidate documents whose hit count differs from Top1 by less than 1 (i.e., multiple potentially similar documents) are collected and proceeded to disambiguation.

[0048] S1023. When the number of candidate documents in the candidate document set exceeds a preset threshold or there are similar documents, determine whether the target document can be identified based on the consistency characteristics between the candidate citation and the candidate document. The consistency characteristics include at least one of the following: title, author, year, and publication location.

[0049] In this embodiment, the preset threshold can be understood as a configurable value used to determine whether the number of candidate documents is excessive (e.g., more than 3). Similar documents can be understood as documents with similar titles, similar authors, or similar years, which can easily lead to ambiguity. Consistency features refer to the degree of matching between the fields (title, author, year, publication location) extracted from the original input candidate citations and the corresponding fields in the candidate documents, such as title edit distance, author spelling consistency, whether the year is the same, whether the publication location is the same, etc. Candidate citations are citation information of the claim text extracted from the original input and are the basis for constructing the candidate citation list. Candidate documents are the basis for constructing the candidate document set. Determining whether the target document can be identified based on consistency features means evaluating the field matching between each candidate document and the candidate citation according to preset priority or weighting rules. If a candidate document is highly matched in all consistency features (e.g., title edit distance is less than the threshold, authors are completely identical, year is the same, publication location is the same), then the candidate document can be uniquely identified as the target document; if multiple candidate documents match in some features but cannot be uniquely distinguished, then they are considered as not uniquely identified.

[0050] Specifically, the process begins by extracting the title, author, year, and publication location information from the candidate citations and comparing them with the corresponding fields of each document in the candidate literature set. For example, it calculates the edit distance of the title, checks for author consistency (spelling edit distance less than a preset threshold), compares the year for equality, and checks the publication location (journal or conference name) for similarity. Different weights are assigned to each field based on their matching degree (e.g., title matching has the highest weight, followed by author and year), and a comprehensive matching score is calculated. If a candidate document's matching score is significantly higher than other candidate documents (e.g., the score difference is greater than a set threshold), that candidate document is identified as the target document; if multiple candidate documents have similar scores, they are considered unable to be uniquely identified.

[0051] S1024. If a unique candidate document can be obtained based on the consistency characteristics, the candidate document shall be determined as the target document.

[0052] In this embodiment, a uniquely determined candidate document can be understood as one that can be clearly selected from the candidate document set based on at least one consistency feature among title, author, year, and publication location, and there are no other candidate documents with similar matching degree.

[0053] Specifically, if, after consistency feature comparison, a candidate document has a title edit distance less than a preset threshold, and its author, year, and publication location are consistent with the candidate citation, then this candidate document is directly identified as the target document, and the matching method is recorded as "matching based on consistency features". If the title alone can uniquely identify the document (e.g., titles are completely identical while other candidate document titles differ significantly), the target document can also be directly identified. After identifying the target document, the subsequent metadata acquisition and consistency verification step S1026 is performed.

[0054] S1025. If the candidate document cannot be uniquely determined based on the consistency feature, the candidate citations in the original input and the metadata of all candidate documents in the candidate document set are input into the large language model. The large language model is instructed to select the most matching candidate document as the target document. If the large language model can uniquely determine the target document, the uniquely determined candidate document is determined as the target document. If the large language model still cannot uniquely determine the target document, an ambiguity prompt and a verification conclusion of questionable existence verification result are generated. The ambiguity prompt includes the ambiguity hit identifier and the candidate document set.

[0055] In this embodiment, the Large Language Model (LLM) refers to a deep learning model trained on a large-scale corpus, capable of understanding natural language and executing instructions. The LLM takes candidate cited text (raw, unnormalized citation strings) and metadata (including title, author, year, publication location, etc.) of all documents in the candidate literature set as input, instructing it to select the best-matching document. If the LLM outputs a unique document number or title with a confidence level higher than a set threshold (e.g., 0.8), it is considered uniquely identifiable; if the LLM outputs multiple candidates or indicates that it cannot be determined, it is considered still not uniquely identifiable. The ambiguity indicator is structured information containing a ambiguity marker and a candidate literature set (listing all potentially matching documents) for subsequent manual review or further processing. The ambiguity marker can be understood as a structured code or label used to distinguish different types of ambiguity states, indicating situations where, during existence verification or opinion validation, multiple similar documents, field conflicts, insufficient information, or other reasons prevent unique identification of the target document or a clear conclusion from being reached.

[0056] Specifically, when consistency feature disambiguation fails, the candidate citation text from the original input and the metadata (organized in list form) of all documents in the candidate document set are concatenated into a prompt word and fed into the large language model. For example, the instruction is: "Please select the best matching document from the following candidate documents based on the following original citation information: [original citation text], and only output the document number. Candidate documents: 1. Title...Author...Year...; 2. Title...;...". After the LLM returns the result, its output is parsed. If a valid document number is returned and that number corresponds to a document in the candidate document set, then the document is identified as the target document, and the hit method is recorded as "Large Language Model Disambiguation Hit". If the content returned by the LLM cannot be parsed into a unique document (e.g., output "Cannot be determined" or lists multiple numbers), it is determined that it still cannot be uniquely determined. At this time, an ambiguity prompt is generated, including the ambiguity hit identifier and the complete candidate document set (i.e., all documents in the candidate document set), and the verification conclusion of the existence check result is marked as questionable, the existence check is terminated, and subsequent metadata acquisition and consistency checks are no longer performed. The questionable result can be returned to the user or the upper-level module, prompting that manual intervention is required for verification.

[0057] S1026. After identifying the target document, obtain the target document's metadata through the document retrieval interface, and perform consistency verification between the metadata and the citation information of the candidate citations to generate an existence verification result. The existence verification result includes the verification conclusion, the target document's metadata, the hit method, the confidence level, and the reason code. The reason code includes at least one of the following: non-existent, unsearchable, hit ambiguity, field inconsistency, or successful match.

[0058] In this embodiment, the document retrieval interface can be an external document retrieval interface (public academic database interface) or an internal document database query interface. Metadata includes standardized fields such as the document's title, author, publication year, publication venue (journal / conference), volume, issue, page number, and abstract. Consistency verification can be understood as comparing the retrieved metadata with the corresponding fields (such as year, author, title) of the citation information in the original input, calculating the edit distance between fields or whether they are completely consistent to determine if there are field conflicts. The existence verification result is a structured output, including the verification conclusion, the target document's metadata, the hit method, the confidence level, and the reason code. The verification conclusion is the final qualitative assessment of the existence verification, including three states: passed (indicating that the candidate citation successfully matched a unique target document with consistent metadata), failed (indicating that no matching documents could be retrieved, or the search results seriously conflicted with the input information), and questionable (indicating that multiple similar documents were retrieved but could not be uniquely determined, or that some fields of the metadata were inconsistent but not completely negated). The hit method describes the type of identifier used for a successful retrieval, such as exact match by DOI, fuzzy match by title, or match by combined identifiers. The confidence level is a value between 0 and 1, reflecting the degree of certainty about the current verification result. The reason code is a specific supplement to the verification result status, including non-existent (no record of the document in the database), unsearchable (interface unavailable or query timed out), ambiguous hit (multiple similar documents), inconsistent fields (such as year or author not matching metadata), or successful match (all fields are consistent and a unique match).

[0059] Specifically, after identifying the target document, the strong identifiers (such as DOI) in the candidate citations or the unique identifier of the target document obtained through retrieval are used as query conditions. The document retrieval interface is then called to obtain the standardized metadata of the target document, including title, author, year, publication location, abstract, and identifier. The obtained metadata is then compared with the original citation information in the candidate citations, performing a field-by-field consistency check. This includes comparing whether the year is consistent, whether the author spelling is similar (edit distance less than a preset threshold), and whether the title keywords match. Based on the verification results, an existence verification result is generated. This result includes the verification conclusion (pass, fail, or questionable), the target document's metadata, the hit method, the confidence level (e.g., 1.0 for exact match via DOI, possibly 0.85 for fuzzy match), and the reason code (e.g., successful match, field inconsistency, ambiguous hit, non-existent, or unsearchable). The hit method includes not only exact match via strong identifiers, match via combined identifiers, and fuzzy match via weak identifiers, but also match via large language model disambiguation. Finally, the existence verification result is passed to the opinion verification module as input for subsequent claim-evidence alignment.

[0060] For example, if a unique match is found and all key fields are consistent, the verification conclusion is passed, and the reason code is successful match; if the search result is empty or the interface is unavailable, the verification conclusion is failed, and the reason code is non-existent or unsearchable; if multiple similar documents exist or the year and author are inconsistent, the verification conclusion is questionable, and the reason code is ambiguous or inconsistent fields; if the target document is selected after disambiguation using a large language model and the verified fields are consistent, the verification conclusion is passed, the hit method is hit by disambiguation using a large language model, the confidence level can be set to 0.7 (adjustable), and the reason code is successful match.

[0061] As a third optional embodiment of this example, Figure 5 This is a flowchart of a reference verification method provided in an embodiment of the present invention, involving viewpoint verification, such as... Figure 2 and Figure 5 As shown, based on the viewpoint citation pairs, semantic alignment and support determination are performed on the claim text and evidence fragments in the target document to generate viewpoint verification conclusions, including: S1031. For each viewpoint citation pair, obtain the first evidence fragment of the target document corresponding to the viewpoint citation pair. The first evidence fragment includes at least an abstract or structured fields.

[0062] In this embodiment, the first piece of evidence can be understood as lightweight evidence quickly obtained from the target document for preliminary support determination, including at least the document's abstract or structured fields (such as research purpose, conclusions, methods, etc.).

[0063] For each opinion citation pair, the claim text, the context to which the claim is tied, and the target document output from the existence check are combined into a single verification input unit. Based on this, keywords, entities, or numerical information are extracted from the claim text and supplementary search terms are generated by combining them with the context, forming query conditions that can be used for full-text search or vector search.

[0064] Specifically, for the current viewpoint citation pair, the first step is to determine whether preliminary verification can be directly performed using lightweight evidence. If the abstract or structured fields of the target document (such as research objective, methods, and conclusion paragraphs) already exist or can be quickly obtained through a literature search interface, this information is used as the first piece of evidence. A verification request is then constructed using the query criteria and the first piece of evidence as input.

[0065] S1032. Input the claim text and the first evidence fragment in the opinion citation pair into the judgment model for classification and judgment, and determine the first verification result of the opinion citation pair. The first verification result includes support, non-support, or doubt. The judgment model includes at least one of the following: large language model, rule-based and similarity matching model, natural language reasoning model, or multi-model voting system.

[0066] In this embodiment, the judgment model can be understood as an algorithm or model used to classify and judge the semantic relationship between the claim text and the first evidence fragment (such as a summary or structured field). The first verification result can be understood as a preliminary supporting judgment conclusion based on the first evidence fragment of the target document.

[0067] Specifically, after obtaining the first piece of evidence, a rapid path verification is performed. The claim text in the current viewpoint citation pair is concatenated with the first piece of evidence to form an input sequence. To improve the accuracy of the judgment, the context of the claim text (such as the preceding sentence, the following sentence, and the paragraph topic) can be incorporated into the input as supplementary information; alternatively, the query conditions based on the current viewpoint citation pair and the first piece of evidence can be directly concatenated into an input sequence. The input sequence is then fed into a judgment model for classification. The judgment model includes at least one of the following: a large language model, a rule-based and similarity-matching model, a Natural Language Inference (NLI) model, or a multi-model voting system. The model performs a binary (or tri-class) classification task. If the claim text is semantically consistent with the first piece of evidence or can be reasonably inferred, the output is "supported"; if the first piece of evidence explicitly refutes the claim or there is a factual conflict, the output is "not supported"; if the first piece of evidence is insufficient in information, semantically ambiguous, or uncertain, the output is "doubtful". A confidence threshold can also be set; when the output is doubtful but the confidence level is low, it is marked as insufficient evidence. This fast path does not require full-text search or fragment location, has a fast response time, and is suitable for most scenarios where the summary information is sufficient to support the judgment.

[0068] S1033. Generate a structured opinion verification conclusion, which includes the first verification result, the first piece of evidence, and the suggested action. The suggested action includes displaying, downgrading the display, or prompting for manual confirmation.

[0069] In this embodiment, the suggested action can be understood as a prompt for subsequent operation based on the verification result, including display (normal display of the result, usually used when the first verification result is passed), downgrade display (presented in gray or warning style, usually used when the first verification result is questionable), or prompt for manual confirmation (requiring user review, usually used when the first verification result is failed).

[0070] Specifically, a structured opinion verification conclusion is generated based on the initial verification result. This conclusion includes at least the initial verification result (support / no support / questionable), the first piece of evidence (i.e., the summary or structured field text used), and a suggested action. If the result is supportive, the suggested action is to display (normally displaying the verification passed indicator); if the result is no supportive, the suggested action is to downgrade the display (presenting the citation risk in gray or a warning style); if the result is questionable, the suggested action is to prompt for manual confirmation or trigger a deeper path. This structured conclusion is encapsulated as a message, which can be returned to the upper-level business module for direct use or used as input conditions for subsequent deep paths.

[0071] Furthermore, when the evidence is insufficient to support the conclusion, based on the viewpoint citation pairs, semantic alignment and support determination are performed on the claim text and the evidence fragments in the target document to generate a viewpoint verification conclusion, which also includes: S1034. For each opinion citation pair, obtain the full text of the target document corresponding to the opinion citation pair, and use the claim text in the opinion citation pair as the query condition to retrieve the second evidence fragment from the full text. The second evidence fragment is a page number level, paragraph level or sentence level, table cell level, or figure caption level fragment.

[0072] In this embodiment, the query conditions can be understood as the retrieval criteria for evidence fragments, which are keywords, entities, or values ​​extracted from the claim text in the opinion citation pair, and used to retrieve relevant fragments in the full-text index. The second evidence fragment is a more granular piece of evidence recalled from the full text, which can be fragments at the page number level, paragraph level, sentence level, table cell level, or figure caption level.

[0073] Specifically, when the first verification result is questionable, or when the evidence is deemed insufficient to support the conclusion based on the confidence threshold, a deep-path verification judgment is triggered. For each opinion citation pair, the full text of the target document is obtained. If the full text has not yet been downloaded, the original document is retrieved via a PDF download interface or local file storage, and text parsing and indexing are performed (e.g., segmented by page number or paragraph). Subsequently, using the claim text in the opinion citation pair as the query condition, at least one of the inverted index, vector index, or hybrid index is used for retrieval. Specific retrieval methods include keyword matching, vector similarity retrieval, or combinations thereof, to recall second evidence fragments from the full-text index. Second evidence fragments can be not only at the paragraph or sentence level, but also at the table cell or figure caption level to ensure more granular evidence support. During recall, fragments are usually sorted by relevance, and the Top K fragments are selected.

[0074] S1035. Input the claim text, the context of the claim text, and the second evidence fragment into the semantic alignment model for support determination, output the second verification result and the confidence of the second verification result, and generate explanatory information. The explanatory information includes at least one of key sentence highlighting, numerical consistency, and negation relation. The semantic alignment model includes at least one of a pre-trained model fine-tuned by natural language inference, a large language model, a rule-based and similarity-matching model, or a multi-model voting system.

[0075] In this embodiment, the context refers to the preceding and following sentences and paragraph topics in the original input where the claim text appears, used to eliminate semantic ambiguity and handle complex logic such as negation, conditionality, and contrast. The semantic alignment model can be understood as a model capable of determining the logical relationship (implication, contradiction, or neutrality) between the claim text and the evidence fragments and outputting a second verification result. Semantic alignment models include at least one of the following: a pre-trained model fine-tuned by Natural Language Inference (NLI), a Large Language Model (LLM), a rule-based and similarity-matching model, or a multi-model voting system. The second verification result can be understood as the supporting assertion output by the alignment model, including support, non-support, or doubt. Explanatory information supplements the supporting judgment result, including key sentence highlighting (indicating the most relevant sentence in the evidence), numerical consistency (comparing whether the numerical values ​​in the claim and the evidence are consistent), and negation relationships (identifying whether the claim is negated by the evidence), etc.

[0076] Specifically, after recalling the second piece of evidence, the claim text in the current viewpoint reference, the context to which the claim is tied (such as the preceding and following sentences, paragraph topics), and the second piece of evidence are all input into the semantic alignment model. The semantic alignment model can determine the logical relationship (implication, contradiction, neutrality) between the claim and the evidence. The model outputs a second verification result (support, non-support, or questionable) and the confidence level of the result (a probability value between 0 and 1). At the same time, the model generates explanatory information, such as highlighting the key sentences most relevant to the claim in the evidence fragment, comparing whether the numerical values ​​are consistent (such as accuracy numerical matching), and identifying whether there is a negation relationship (such as "not found" or "no significant difference"). This explanatory information enhances the credibility and verifiability of the judgment result.

[0077] S1036. Generate a structured opinion verification conclusion, which includes: a second verification result, a second piece of evidence, evidence location information, a reason code, and a suggested action. The second verification result includes support, non-support, or doubt; the evidence location information includes at least one of page number, paragraph, table cell, and figure caption; the reason code includes insufficient evidence, contradiction, irrelevance, or excessive ambiguity; and the suggested action includes display, downgrade display, or prompt for manual confirmation.

[0078] In this embodiment, evidence location information can be understood as the specific location of the second evidence fragment in the target document, accurate to the page number, paragraph number, table cell, or figure caption number. The reason code is a classification label describing the reason for the verification conclusion, which in the deep path includes insufficient evidence (the recalled fragment does not contain enough information), contradiction (the evidence directly conflicts with the claim), irrelevance (the recalled fragment is unrelated to the claim's theme), or excessive ambiguity (the evidence can be interpreted in multiple ways). Confidence level is a probability value output by the model, representing the degree of certainty regarding the verification conclusion.

[0079] Specifically, based on the second verification result and its corresponding confidence level output by the alignment model, combined with the second evidence fragment (the specific recalled text), evidence location information (accurate to page number, paragraph number, table cell or figure caption number), reason code (insufficient evidence, contradiction, irrelevance, or excessive ambiguity), and suggested actions (display, downgrade display, or prompt for manual confirmation), a structured opinion verification conclusion is generated as the final result of the deep path. Evidence location information allows users to directly jump to the corresponding location in the original text for verification; reason codes help upper-level business modules understand the specific reasons behind the conclusion; and suggested actions guide the product side in differentiated display. This structured conclusion maintains the same format as the fast path output, facilitating unified processing.

[0080] Understandably, when a claim needs to be supported by multiple documents, the claim can be verified separately against multiple viewpoints, and the results of each verification can be combined to output a joint judgment. For example, S1031-S1036 can be performed independently on each target document, and the final viewpoint verification conclusion can be obtained through voting or weighting.

[0081] The method provided in this invention is not only applicable to question-and-answer scenarios, but can also be applied to other uses such as citation quality control in review writing, evidence verification in knowledge base construction, and fact verification of scientific research communication materials.

[0082] In one embodiment, Figure 6 This is a schematic diagram of a reference verification device provided in an embodiment of the present invention. Figure 6 As shown, the device includes: The input preprocessing module 21 is used to acquire the raw input, extract the claim text to be verified and the candidate citation list associated with the claim text from the raw input, and form structured citation information and opinion citation pairs; The existence verification module 22 is used to perform existence verification on each candidate citation in the candidate citation list based on the structured citation information, so as to determine whether the target document corresponding to the candidate citation actually exists and can be obtained, and generate an existence verification result. The existence verification result includes at least a reason code and / or a hit method. The opinion verification module 23 is used to perform semantic alignment and support determination on the claim text and the evidence fragments in the target document based on the opinion citation pair, and generate an opinion verification conclusion. The opinion verification conclusion includes at least the suggested action and / or evidence location information.

[0083] The reference verification device used in this technical solution automates the entire reference verification process, improving the accuracy, traceability, and engineering reusability of reference verification.

[0084] Optionally, the existence verification module 22 is specifically used for: The candidate citations are retrieved according to the priority of the citation identifiers to determine the candidate document set corresponding to each candidate citation; When the candidate document set is uniquely matched, the uniquely matched candidate document is determined as the target document; When the number of candidate documents in the candidate document set exceeds a preset threshold or there are similar documents, it is determined whether the target document can be identified based on the consistency characteristics between the candidate citations and the candidate documents. The consistency characteristics include at least one of title, author, year, and publication location. If a unique candidate document can be obtained based on the aforementioned consistency characteristics, the candidate document is determined as the target document. If the candidate document cannot be uniquely determined based on the consistency feature, the candidate citations in the original input and the metadata of all candidate documents in the candidate document set are input into the large language model. The large language model is instructed to select the most matching candidate document as the target document. If the large language model can uniquely determine the target document, the uniquely determined candidate document is determined as the target document. If the large language model still cannot uniquely determine the target document, an ambiguity prompt and a verification conclusion of questionable existence verification result are generated. The ambiguity prompt includes a hit ambiguity identifier and a candidate document set. After identifying the target document, the metadata of the target document is obtained through the document retrieval interface, and the consistency of the metadata with the citation information of the candidate citation is verified to generate an existence verification result. The existence verification result includes the verification conclusion, the metadata of the target document, the hit method, the confidence level and the reason code. The reason code includes at least one of non-existent, unsearchable, hit ambiguity, field inconsistency or successful match.

[0085] Optionally, the citation identifier includes strong identifiers, weak identifiers, and combined identifiers. The strong identifier includes at least a location identifier, the weak identifier includes a title, and the combined identifier includes at least a combination of title and author, a combination of author and year, and a combination of author and publication location.

[0086] Optionally, the opinion verification module 23 is specifically used for: For each pair of viewpoint citations, obtain the first evidence fragment of the target document corresponding to the pair of viewpoint citations, wherein the first evidence fragment includes at least an abstract or structured fields; The claim text in the opinion citation pair and the first evidence fragment are input into the judgment model for classification and judgment to determine the first verification result of the opinion citation pair. The first verification result includes support, non-support, or doubt. The judgment model includes at least one of the following: a large language model, a rule-based and similarity matching model, a natural language reasoning model, or a multi-model voting system. Generate a structured opinion verification conclusion, which includes the first verification result, the first evidence fragment, and a suggested action, including displaying, downgrading the display, or prompting for manual confirmation.

[0087] Optionally, the opinion verification module 23 is specifically used for: For each pair of viewpoint citations, the full text of the target document corresponding to the pair of viewpoint citations is obtained. Using the claim text in the pair of viewpoint citations as the query condition, a second piece of evidence is retrieved from the full text. The second piece of evidence is a page-level, paragraph-level or sentence-level, table-cell-level, or figure-caption-level piece of evidence. The assertion text in the opinion reference pair, the context of the assertion text, and the second evidence fragment are input into the semantic alignment model for support determination. The model outputs the second verification result and the confidence level of the second verification result, and generates explanatory information. The explanatory information includes at least one of key sentence highlighting, numerical consistency, and negation relation. The semantic alignment model includes at least one of a pre-trained model fine-tuned by natural language inference, a large language model, a rule-based and similarity-matching model, or a multi-model voting system. A structured opinion verification conclusion is generated, which includes: a second verification result, a second piece of evidence, evidence location information, a reason code, and a suggested action; wherein, the second verification result includes support, non-support, or doubt; the evidence location information includes at least one of page number, paragraph, table cell, and figure caption; the reason code includes insufficient evidence, contradiction, irrelevant, or excessive ambiguity; and the suggested action includes display, downgrade display, or prompt for manual confirmation.

[0088] Optionally, the input preprocessing module 21 is specifically used for: Determine whether the original input is academic text; if not, output an inapplicable result. If so, the corresponding extraction strategy is executed according to the carrier type of the original input to obtain the claim text to be verified and the candidate citation list associated with the claim text, forming structured citation information, wherein the carrier type includes files, links or text; When the opinion extraction conditions are met, opinion citation pairs between the claim text and the citations in the candidate citation list are extracted.

[0089] The reference verification device provided in the embodiments of the present invention can execute the reference verification method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0090] In one embodiment, Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. For example... Figure 7 The diagram illustrates a schematic representation of an electronic device 10 that can be used to implement embodiments of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0091] like Figure 7 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded into the RAM 13 from storage unit 18. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0092] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0093] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the reference verification method.

[0094] In some embodiments, the reference verification method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the reference verification method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the reference verification method by any other suitable means (e.g., by means of firmware).

[0095] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0096] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0097] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0098] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0099] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0100] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0101] This invention also provides a computer program product, including a computer program that, when executed by a processor, can implement the reference verification method provided in any embodiment of this application.

[0102] In the implementation of the computer program product, computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0103] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0104] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for verifying references, characterized in that, include: Obtain the original input, extract the claim text to be verified and the candidate citation list associated with the claim text from the original input, and form structured citation information and opinion citation pairs; Based on the structured citation information, an existence check is performed on each candidate citation in the candidate citation list to determine whether the target document corresponding to the candidate citation actually exists and can be obtained, and an existence check result is generated. The existence check result includes at least a reason code and / or a hit method. Based on the aforementioned viewpoint citation pairs, semantic alignment and support determination are performed on the assertion text and evidence fragments in the target document to generate viewpoint verification conclusions. The viewpoint verification conclusions include at least suggested actions and / or evidence location information.

2. The method according to claim 1, characterized in that, The step of performing an existence check on each candidate citation in the candidate citation list based on the structured citation information includes: The candidate citations are retrieved according to the priority of the citation identifiers to determine the candidate document set corresponding to each candidate citation; When the candidate document set is uniquely matched, the uniquely matched candidate document is determined as the target document; When the number of candidate documents in the candidate document set exceeds a preset threshold or there are similar documents, it is determined whether the target document can be identified based on the consistency characteristics between the candidate citations and the candidate documents. The consistency characteristics include at least one of title, author, year, and publication location. If a unique candidate document can be obtained based on the aforementioned consistency characteristics, the candidate document is determined as the target document. If the candidate document cannot be uniquely determined based on the consistency feature, the candidate citations in the original input and the metadata of all candidate documents in the candidate document set are input into the large language model. The large language model is instructed to select the most matching candidate document as the target document. If the large language model can uniquely determine the target document, the uniquely determined candidate document is determined as the target document. If the large language model still cannot uniquely determine the target document, an ambiguity prompt and a verification conclusion of questionable existence verification result are generated. The ambiguity prompt includes a hit ambiguity identifier and a candidate document set. After identifying the target document, the metadata of the target document is obtained through the document retrieval interface, and the consistency of the metadata with the citation information of the candidate citation is verified to generate an existence verification result. The existence verification result includes the verification conclusion, the metadata of the target document, the hit method, the confidence level and the reason code. The reason code includes at least one of non-existent, unsearchable, hit ambiguity, field inconsistency or successful match.

3. The method according to claim 2, characterized in that, The citation identifiers include strong identifiers, weak identifiers, and combined identifiers. The strong identifiers include at least a location identifier, the weak identifiers include a title, and the combined identifiers include at least a combination of title and author, a combination of author and year, and a combination of author and publication location.

4. The method according to claim 1, characterized in that, The step of semantically aligning and determining the support of the claim text with the evidence fragments in the target document based on the aforementioned viewpoint citation pair, and generating a viewpoint verification conclusion, includes: For each pair of viewpoint citations, obtain the first evidence fragment of the target document corresponding to the pair of viewpoint citations, wherein the first evidence fragment includes at least an abstract or structured fields; The claim text in the opinion citation pair and the first evidence fragment are input into the judgment model for classification and judgment to determine the first verification result of the opinion citation pair. The first verification result includes support, non-support, or doubt. The judgment model includes at least one of the following: a large language model, a rule-based and similarity matching model, a natural language reasoning model, or a multi-model voting system. Generate a structured opinion verification conclusion, which includes the first verification result, the first evidence fragment, and a suggested action, including displaying, downgrading the display, or prompting for manual confirmation.

5. The method according to claim 1, characterized in that, The step of semantically aligning and determining the support of the claim text with the evidence fragments in the target document based on the aforementioned viewpoint citation pair, and generating a viewpoint verification conclusion, includes: For each pair of viewpoint citations, the full text of the target document corresponding to the pair of viewpoint citations is obtained. Using the claim text in the pair of viewpoint citations as the query condition, a second piece of evidence is retrieved from the full text. The second piece of evidence is a page-level, paragraph-level or sentence-level, table-cell-level, or figure-caption-level piece of evidence. The assertion text in the opinion reference pair, the context of the assertion text, and the second evidence fragment are input into the semantic alignment model for support determination. The model outputs the second verification result and the confidence level of the second verification result, and generates explanatory information. The explanatory information includes at least one of key sentence highlighting, numerical consistency, and negation relation. The semantic alignment model includes at least one of a pre-trained model fine-tuned by natural language inference, a large language model, a rule-based and similarity-matching model, or a multi-model voting system. A structured opinion verification conclusion is generated, which includes: a second verification result, a second piece of evidence, evidence location information, a reason code, and a suggested action; wherein, the second verification result includes support, non-support, or doubt; the evidence location information includes at least one of page number, paragraph, table cell, and figure caption; the reason code includes insufficient evidence, contradiction, irrelevant, or excessive ambiguity; and the suggested action includes display, downgrade display, or prompt for manual confirmation.

6. The method according to claim 1, characterized in that, The process of obtaining the original input, extracting the claim text to be verified and the candidate citation list associated with the claim text from the original input, and forming structured citation information and opinion citation pairs includes: Determine whether the original input is academic text; if not, output an inapplicable result. If so, the corresponding extraction strategy is executed according to the carrier type of the original input to obtain the claim text to be verified and the candidate citation list associated with the claim text, forming structured citation information, wherein the carrier type includes files, links or text; When the opinion extraction conditions are met, opinion citation pairs between the claim text and the citations in the candidate citation list are extracted.

7. A reference verification device, characterized in that, include: The input preprocessing module is used to acquire the raw input, extract the claim text to be verified and the candidate citation list associated with the claim text from the raw input, and form structured citation information and opinion citation pairs; An existence verification module is used to perform an existence verification on each candidate citation in the candidate citation list based on the structured citation information, so as to determine whether the target document corresponding to the candidate citation actually exists and can be obtained, and generate an existence verification result. The existence verification result includes at least a reason code and / or a hit method. The opinion verification module is used to perform semantic alignment and support determination on the claim text and the evidence fragments in the target document based on the opinion citation pair, and generate an opinion verification conclusion. The opinion verification conclusion includes at least the suggested action and / or evidence location information.

8. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform a reference verification method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute and implement a reference verification method according to any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements a reference verification method according to any one of claims 1-6.

Images