A citation post-processing correction method and related apparatus

Abstract

This application discloses a post-citation correction method and related apparatus, relating to the field of information retrieval technology. The method decomposes a citation-containing answer into at least one factual text. A target re-ranking model is used to determine the support score between each factual text and each document in the target document set. The support scores of factual texts and documents with citation relationships in the citation-containing answer are used as the first support score, and the remaining support scores are used as the second support score. Prior enhancement processing is applied to the first support score to obtain the corresponding enhanced score. Based on the enhanced score and the second support score, valid citation documents corresponding to each factual text are selected from the target document set. Based on the valid citation documents, a citation correction answer corresponding to the citation-containing answer is generated. This application, based on deep semantic scoring and prior knowledge enhancement using a target re-ranking model, achieves the filtering of erroneous citations and the supplementation of missing citations, thus improving the accuracy of citation correction.

Description

Technical Field

[0001] This application relates to the field of information retrieval technology, and in particular to a post-citation processing correction method and related apparatus. Background Technology

[0002] In recent years, Large Language Models (LLMs) have made groundbreaking progress in natural language understanding and generation. Retrieval-Augmented Generation (RAG), a technological paradigm that combines information retrieval with text generation, indexes knowledge bases, retrieves documents relevant to user queries, and instructs the LLM to generate citation-annotated answers based on the retrieved content. It has been widely applied in fields such as intelligent search, question answering, and automatic research report generation.

[0003] However, in practical applications of RAG systems, LLMs exhibit significant shortcomings in source attribution of cited documents. Industry research indicates that the citation accuracy of mainstream generative search engines is only around 74%. System audits of multiple LLMs revealed that approximately 80% of "unverifiable facts" were not purely hallucinations, but rather citation errors—meaning the LLM could correctly extract factual information from the retrieved documents, but attributed it to the wrong references. This inaccurate citation problem not only reduces the operability of responses but also severely damages users' trust in AI-generated content.

[0004] To address the issue of citation error correction, existing technologies have proposed several correction methods. For example, prompting engineering-based methods require the LLM to output citations in the generation instructions, but this method can only guide the LLM to cite as many citations as possible during generation and cannot correct erroneous citations afterward. Keyword matching-based citation correction methods assess citation relevance by calculating the lexical overlap between factual text in the LLM's response and candidate documents, but this method can only be used to verify the correctness of already cited citations and cannot proactively supplement the factual text with appropriate citation sources.

[0005] Therefore, how to identify and delete erroneous citations while actively matching and adding correct citation documents to factual texts has become a pressing technical problem that needs to be solved. Summary of the Invention

[0006] In view of the above problems, this application provides a post-citation processing correction method and related apparatus to achieve the purpose of both identifying and deleting erroneous citations and actively supplementing factual texts with correct citation documents. The specific solution is as follows: The first aspect of this application provides a post-citation processing correction method, including: Obtain the cited answer and target document set generated by the target large language model, decompose the cited answer into at least one fact text, and the target document set includes all documents cited in the cited answer; A pre-trained target re-ranking model is used to determine the support score between each fact text and each document in the target document set, whereby the support score characterizes the degree to which a document supports the fact text. The support score of the factual text and document with citation relationship in the cited answer is taken as the first support score, and the remaining support scores are taken as the second support score. The first support score is subjected to a priori enhancement processing to obtain the enhanced score corresponding to the first support score. The enhanced score is higher than the corresponding first support score. Based on the enhanced score corresponding to the first support score and the second support score, valid citation documents corresponding to each factual text are selected from the target document set, and a citation-corrected answer corresponding to the citation answer is generated based on the valid citation documents.

[0007] In one possible implementation, the step of filtering valid citation documents corresponding to each fact text from the target document set based on the enhancement score corresponding to the first support score and the second support score includes: The enhancement score corresponding to the first support score and the second support score are grouped according to the fact text to obtain a group corresponding to each fact text, wherein each enhancement score and each second support score in the group are used as reference scores; For each of the aforementioned fact texts, process them separately: For documents in the target document set that are not referenced by the fact text, based on the grouping corresponding to the fact text, documents with a reference score greater than a preset insertion threshold are selected from the unreferenced documents and used as the first candidate documents corresponding to the fact text. For the documents in the target document set whose factual text has been referenced, based on the grouping corresponding to the factual text, documents with a reference score greater than a preset filtering threshold are selected from the referenced documents and used as the second candidate documents corresponding to the factual text. From the first and second candidate documents corresponding to each fact text, select at most m candidate documents with the highest reference scores as the valid citation documents corresponding to each fact text, where m is a positive integer.

[0008] In one possible implementation, before filtering valid citation documents corresponding to each fact text from the target document set based on the enhancement score corresponding to the first support score and the second support score, the method further includes: For target fact text that does not cite any documents, determine whether the character length of the target fact text is greater than a preset length threshold. If the determination result is negative, do not perform the step of filtering valid citation documents corresponding to each fact text from the target document set based on the enhanced score corresponding to the first support score and the second support score.

[0009] In one possible implementation, the insertion threshold and the filtering threshold are respectively used as target thresholds. The process of determining the target thresholds includes: Obtain a validation set, wherein each sample in the validation set includes a first factual text sample, a document sample, and a real label. The real label corresponding to the insertion threshold is used to indicate whether the document sample should be inserted for the first factual text sample. The real label corresponding to the filtering threshold is used to indicate whether the document sample truly supports the first factual text sample. For each candidate threshold in the preset candidate threshold set, each sample in the validation set is classified according to the candidate threshold to obtain the predicted classification result corresponding to each sample. Based on the predicted classification results corresponding to all samples in the validation set and the true label, the classification performance index corresponding to the candidate threshold is generated. Based on the classification performance index corresponding to each candidate threshold, the objective function value corresponding to each candidate threshold is determined, and based on the objective function values ​​corresponding to all candidate thresholds in the candidate threshold set, the target threshold is determined.

[0010] In one possible implementation, determining the target threshold based on the objective function values ​​corresponding to all candidate thresholds in the candidate threshold set includes: From the candidate threshold set, the candidate threshold with the largest corresponding objective function value is selected as the basic threshold, wherein the objective function value corresponding to the basic threshold is the basic function value; From the set of candidate thresholds, determine the candidate thresholds that are numerically adjacent to the base threshold; The target threshold is obtained by averaging the thresholds whose corresponding objective function values ​​are above a preset percentage of the basic function value among the adjacent candidate thresholds and the basic threshold.

[0011] In one possible implementation, the type of the classification performance metric is determined based on the query scenario corresponding to the cited answer.

[0012] In one possible implementation, determining the support score of each of the fact texts and each document in the target document set using a pre-trained target re-ranking model includes: Each fact text and each document are input into the target reordering model to obtain the log probabilities of the first target token and the second target token generated by the model. The first target token indicates that the document supports the fact text, and the second target token indicates that the document does not support the fact text. The support score is calculated based on the logarithmic probabilities of the first target token and the second target token, respectively.

[0013] In one possible implementation, the step of performing prior augmentation processing on the first support score to obtain the augmentation score corresponding to the first support score includes: A preset prior reward value is added to the first support score to obtain the enhanced score corresponding to the first support score.

[0014] In one possible implementation, the training process of the target reordering model includes: Acquire training data, which includes a second factual text sample and corresponding positive example documents and multiple negative example documents. The positive example documents are those that actually support the second factual text sample, and the negative example documents are those that do not support the second factual text sample. The support scores of the second factual text sample with the positive example document and each of the negative example documents are determined by a pre-built initial re-ranking model. Based on the support scores of the second factual text sample and the positive and negative example documents respectively, a contrastive learning loss is generated; The initial re-ranking model is trained using the contrastive learning loss to obtain the target re-ranking model.

[0015] In one possible implementation, decomposing the cited answer into at least one factual text includes: The citation markers in the citation-containing response are identified and removed using a preset regular expression to obtain a citation-free response. The uncited answer was segmented into sentences to obtain multiple sentence texts; Sentences that meet preset conditions from the plurality of sentence texts are removed, and each remaining sentence text is treated as a fact text to obtain at least one fact text. The preset conditions include at least one of the following: the sentence text is a title sentence, the sentence text is a quotation sentence in chart format or other preset format, or the sentence text is a Chinese numbered title.

[0016] In one possible implementation, generating the citation-corrected answer corresponding to the citation-bearing answer based on the valid citation document includes: Based on the valid citation document corresponding to each fact text, generate a citation-corrected fact text corresponding to each fact text; Obtain the line number of each fact text in the cited response, concatenate the citation correction fact texts belonging to the same line in all the citation correction fact texts together, and connect the citation correction fact texts belonging to different lines with a newline character to obtain the initial correction response; The initial corrected response is obtained by compressing multiple consecutive newline characters into the target number.

[0017] A second aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the post-citation correction method of the first aspect or any implementation thereof.

[0018] A third aspect of this application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the post-citation correction method of the first aspect or any implementation thereof.

[0019] A fourth aspect of this application provides a computer storage medium carrying one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the post-citation correction method of the first aspect or any implementation thereof.

[0020] By employing the aforementioned technical solution, the post-citation correction method provided in this application obtains the cited responses and target document set generated by the target large language model. Considering that large language models typically annotate citations on a fact-by-fact basis when generating cited responses, this application first decomposes the cited responses into at least one factual text to ensure that the granularity of subsequent citation correction remains consistent with the original generation granularity. Next, a pre-trained target re-ranking model is used to determine the support score of each factual text and each document in the target document set, objectively quantifying the degree to which a document supports the factual text. Because this application can leverage the deep semantic interaction capabilities of the re-ranking model to calculate a support score based on factual support, subsequent citation filtering using the support score as a selection criterion can effectively distinguish between documents that truly support the factual text and semantically similar but actually unsupportive difficult negative samples, thereby significantly improving the accuracy and reliability of citation filtering.

[0021] Considering that the existing factual and document citation relationships in the original cited responses often contain strong prior information—that is, the documents cited by the target large language model and the corresponding factual texts usually have a high correlation—this application uses the support scores of factual texts and documents with citation relationships in the cited responses as the first support score and the remaining support scores as the second support score. The first support score is then enhanced to be higher than the original first support score. By proactively improving the support scores of the documents cited by the target large language model, this application can effectively compensate for the potential underestimation of cited documents by the target re-ranking model. Finally, based on the enhanced score and the second support score corresponding to the first support score, valid citation documents corresponding to each factual text are selected from the target document set. Based on these valid citation documents, a citation correction response corresponding to the cited response is generated. This not only eliminates erroneous citation documents that cannot effectively support the factual text but also proactively matches and adds correct citation documents that were not originally cited but have supporting functions, significantly improving the completeness and accuracy of citation correction. Attached Figure Description

[0022] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0023] Figure 1 A schematic diagram of a system architecture provided for this application; Figure 2 A flowchart illustrating a post-citation correction method provided for this application; Figure 3 A schematic diagram of the post-citation processing and correction system provided in this application; Figure 4 A schematic diagram of a post-citation correction device provided in this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0024] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0025] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0026] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0027] This application provides a post-citation processing correction method and related apparatus, which can efficiently correct citation annotation errors in RAG systems through post-processing without modifying the LLM generation process.

[0028] Optionally, the post-citation processing correction method and related apparatus provided in this application can be applied to scenarios such as intelligent search, knowledge question answering, automatic generation of research reports, and verification of academic literature citations.

[0029] It should be noted that the above scenarios are merely examples and are not intended to limit this application.

[0030] Optionally, the post-citation correction method provided in this application can be applied to, for example... Figure 1 The system architecture shown includes a terminal 100 and a server 200. The server 200 may include one or more servers (…). Figure 1 (This example uses a server as an illustration).

[0031] Either terminal 100 or server 200 can be used independently to execute the post-citation correction method provided in the embodiments of this application. Alternatively, terminal 100 and server 200 can also be used collaboratively to execute the post-citation correction method provided in the embodiments of this application.

[0032] For example, terminal 100 collects or receives a set of citation-bearing responses and target documents, then sends the citation-bearing responses and target document set to server 200 for subsequent citation correction processing, and returns the citation-corrected responses to terminal 100; or, after obtaining the citation-bearing responses and target document set, terminal 100 decomposes the citation-bearing responses into at least one factual text, then sends the at least one factual text and the target document set to server 200, server 200 performs citation correction processing to obtain the valid citation documents corresponding to each factual text, and sends them back to terminal 100, terminal 100 then generates the citation-corrected responses corresponding to the citation-bearing responses based on the valid citation documents.

[0033] Of course, the above implementation is merely an example and is not intended to limit this application.

[0034] The following description Figure 1 The product form of the mid-terminal 100; The terminal 100 in this application embodiment can be a mobile phone, tablet computer, wearable device, vehicle device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc., and this application embodiment does not impose any restrictions on it.

[0035] In practical applications, users can input query requests through any of the above product forms and receive citation-included answers generated by the target large language model. Then, the citation post-processing correction method provided in this application can be used to correct the citations in the citation-included answers, thereby obtaining a final output with more accurate citations and stronger verifiability.

[0036] It should be noted that the above system architecture and terminal product form are merely examples and are not intended to limit the scope of this application.

[0037] To enable those skilled in the art to better understand this application, the post-citation correction method of the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0038] Reference Figure 2 , Figure 2 This is a flowchart illustrating a post-citation correction method provided in an embodiment of this application, as shown below. Figure 2 As shown, this post-citation correction method may include: Step S201: Obtain the set of cited answers and target documents generated by the target large language model, and decompose the cited answers into at least one fact text.

[0039] In this embodiment, the process of generating a citation-based answer by the target large language model is based on the RAG retrieval process. That is, after receiving a user query, the target large language model first performs a RAG retrieval to obtain a set of relevant documents, and then generates a citation-based answer to the user query based on the set of relevant documents.

[0040] For example, if a user queries "What is photosynthesis?", RAG retrieves relevant documents including Documents 1-5, and the final citation-supported answer is as follows: Photosynthesis is the process by which plants, algae and some bacteria use light energy to convert carbon dioxide and water into organic matter (mainly glucose) and release oxygen[1][2].

[0041] The reaction mainly occurs on the thylakoid membrane of chloroplasts, where the light reaction phase produces ATP and NADPH, while the dark reaction phase (Calvin cycle) fixes carbon dioxide to produce sugars[3].

[0042] External factors affecting the rate of photosynthesis include light intensity, carbon dioxide concentration, temperature and water supply[1][4].

[0043] Figure 1 A schematic diagram showing the specific steps of the light and dark reactions in photosynthesis[^5]”.

[0044] It should be noted that the citation-tagged responses in this application are essentially query responses with citation tags. For example, the first sentence cites the text content of document 1 and document 2, marked as “[1]” and “[2]”, the second sentence cites the text content of document 3, marked as “[3]”, the third sentence cites the text content of document 1 and document 4, marked as “[1]” and “[4]”, and the fourth sentence cites the chart content of document 5, marked as “[^5]”.

[0045] Optionally, the target large language model mentioned above can be any of the following large language models: iFlytek Spark Large Model, Deepin Search Large Model, Doubao Large Model, etc. Of course, the target large language model can also be other types, which are not specifically limited here.

[0046] In this embodiment, the target document set includes all documents cited in the cited answer. For example, the target document set is the relevant document set in the example above.

[0047] In order to perform fine-grained correction of citations, this embodiment needs to first break down the continuous citation-based responses into at least one factual text.

[0048] Optionally, a factual text can be a sentence in a cited response or a block of text that is based on a single fact (a sentence may contain multiple facts, and multiple sentences may contain a single fact).

[0049] Among them, "facts" refer to propositional information expressed in the text that can be objectively verified. A statement constitutes a fact and needs to meet the following three conditions: (1) propositional condition, that is, it expresses a complete proposition that can be analyzed in the form of "subject, predicate, object" or "entity, attribute, value"; (2) verifiability condition, that is, the proposition can be verified or falsified by consulting external source documents; (3) objectivity condition, that is, the expression is in an objective statement tone, rather than a subjective judgment, speculation, suggestion or instruction. In the specific implementation, large language models such as Doubao can be used to perform fact segmentation on the cited answers based on the above three conditions to obtain text blocks containing individual facts. The training data used in the model training stage comes from more than 500 historical cited answers of the model collected and sorted by humans, as well as text blocks of historical cited answers labeled by humans with individual facts as the basic unit.

[0050] Taking a factual text as an example, which is a sentence in a cited answer, in one possible implementation, the process of "decomposing a cited answer into at least one factual text" in this embodiment may include: identifying and removing all citation tags in the cited answer through a preset regular expression to obtain a citationless answer; splitting the citationless answer into sentences to obtain multiple sentence texts; removing sentence texts that meet preset conditions from the multiple sentence texts; and treating each remaining sentence text as a factual text to obtain at least one factual text.

[0051] Specifically, a multi-mode regular expression matcher is first used to remove all citation tags in the cited response. Here, the "multi-mode" configuration enables this application to support multiple citation formats, including but not limited to: [x], [[x]], [[^x]], [^x], [x,y], ^x^, {x}, [[x][y][z]] and other nested and mixed formats.

[0052] For example, in a multi-pattern regular expression matcher, a regular expression can be configured for each citation format, and then all regular expressions can be merged into a single overall regular expression to achieve unified removal of citations throughout the text.

[0053] Next, the plain text answer without citations is divided into multiple independent sentence texts according to the punctuation marks (period, question mark, exclamation mark, etc.) and line breaks of languages ​​such as Chinese and English. Each sentence text is a fact text. At the same time, the line number of each fact text can be recorded to maintain the original text structure during subsequent correction.

[0054] It should be noted that special attention needs to be paid to distinguishing between decimal points and periods when splitting by sentence. For example, the decimal point in the numerical value should be judged by the context to avoid misinterpreting it as the end of the sentence.

[0055] Considering that not all sentence texts are factual statements, and some sentence texts may not contain facts, in order to avoid unnecessary citation filtering and insertion, this embodiment can filter the segmented sentence texts by setting preset conditions, remove the sentence texts that meet the preset conditions, and obtain the remaining sentence texts as factual texts.

[0056] Optionally, the preset conditions include at least one of the following: the sentence text is a title sentence, the sentence text is a quotation sentence in chart format or other preset format, or the sentence text is a Chinese numbered title.

[0057] For example, heading sentences starting with "#" are merely summaries of the content under the heading, making it easier for users to understand the content. Therefore, these heading sentences can be retained directly without citation correction. Similarly, citations of figures or tables, or certain formatted citations, do not require citation correction and can be skipped. Additionally, although Chinese numbered headings (such as I, II, III) contain numbers, these numbers are not citation identifiers and can be skipped directly when Chinese numbered headings are identified, without inserting a citation.

[0058] Through the above processing, the content that needs citation correction (including citation filtering and insertion) can be defined at the finest granularity and the smallest scope, which improves the efficiency of subsequent citation correction.

[0059] In one possible implementation, the above-mentioned processes of removing citation markers and segmenting by sentence can be performed sequentially or simultaneously. For example, citation markers can be identified and removed while sentence segmentation markers are identified and sentence segmentation is performed.

[0060] Considering that the subsequent correction process requires the factual text to correspond with the document it references (i.e., the citation), optionally, this embodiment can also, at the same time as each factual text identifies the citation mark, correspond the citation mark with the factual text, parse the citation mark to obtain the citation identifier (ID), and perform deduplication and format standardization processing for use in the subsequent correction process.

[0061] As mentioned earlier, some citations are identified in a nested format, such as [[^3,^4,6,7]. In this case, a recursive inner-layer parsing strategy can be used to correctly identify each independent citation and accurately correspond it to the factual text.

[0062] In addition, it is necessary to distinguish between text references and chart references (identified by the "^" mark) in order to perform global deduplication tracking of chart references.

[0063] Step S202: Determine the support score of each fact text and each document in the target document set using a pre-trained target re-ranking model.

[0064] The support score represents the degree to which a document supports the facts presented in the factual text. A higher support score indicates that the document's content can factually confirm, corroborate, or deduce the content expressed in the factual text, and thus the document has a higher semantic relevance to the factual text.

[0065] After obtaining each fact text, this application can use a pre-trained target reordering model to calculate a quantitative score, i.e. a support score, for each fact text and each document in the target document set, to characterize the degree of fact support of the document for the corresponding fact text. The higher the support score, the more consistent the document content is with the fact text, and the more effective it is as a source of the fact.

[0066] Optionally, the aforementioned target re-ranking model can employ the Qwen3-Reranker model based on a generative architecture. Qwen3-Reranker is a model in the Qwen3 series specifically designed for text re-ranking tasks. The training data uses publicly available retrieval and re-ranking datasets (such as MS MARCO (a large-scale machine reading comprehension dataset)) as the basic data source, covering over 500 languages ​​and more than 200 domains. The Qwen3-Reranker model consists of multiple stacked Transformers. During the inference phase, factual text (as the query) and documents are concatenated into a complete input sequence. The multi-head self-attention mechanism of the Transformer enables deep semantic interaction at the token level between the query and the document, thereby more accurately capturing the fine-grained semantic relationships between them. Based on these captured semantic relationships, a precise semantic relevance score is generated as the supporting score mentioned above.

[0067] Those skilled in the art will understand that the Qwen3-Reranker model specifically provides three parameter scale configurations: 0.6B (Billion), 4B, and 8B. The 0.6B version contains 28 Transformer layers, while the 4B and 8B versions contain 36 Transformer layers. In this embodiment, any one of the 0.6B, 4B, or 8B versions can be used depending on the actual situation.

[0068] Considering the potentially large number of factual texts and documents in the target document set, the calculation of supporting scores can be time-consuming. To improve computational efficiency, one possible implementation is to deploy the target re-ranking model (such as Qwen3-Reranker-4B, which natively supports a maximum of 32K (32768) token contexts) on a high-throughput, low-latency inference engine like vLLM, utilizing parallel computing and prefix caching of the graphics processing unit (GPU) to accelerate inference. In actual deployment, the input truncation length and model context window can be configured to take values ​​on demand (e.g., configured to several thousand tokens respectively) according to memory and latency requirements, in order to achieve a balance between long document processing capability and resource consumption.

[0069] Step S203: Take the supporting scores of factual texts and documents with citation relationships in the cited answers as the first supporting score, and take the remaining supporting scores as the second supporting scores. Perform a priori enhancement processing on the first supporting score to obtain the enhanced score corresponding to the first supporting score. The enhanced score is higher than the corresponding first supporting score.

[0070] Considering that the cited responses generated by the target large language model inherently contain citation relationships between factual text and documents, although these citation relationships have a certain probability of error, as described in the background, the target large language model can select relevant citations in most cases. Therefore, in order to utilize the strong prior knowledge implied by these citation relationships (the documents cited by the target large language model and the corresponding factual texts usually have a high correlation), this application can use the support scores of the factual texts and documents with citation relationships in the cited responses as the first support score, and perform prior enhancement processing on the first support score so that the enhanced score corresponding to the first support score is higher than the first support score itself, thereby increasing the probability of documents in the citation relationships being selected and reducing the risk of miscorrection caused by relying solely on the support scores.

[0071] In one possible implementation, the process of “performing prior enhancement processing on the first support score to obtain the enhanced score corresponding to the first support score” may include: adding a preset prior reward value to the first support score to obtain the enhanced score corresponding to the first support score.

[0072] For example, the prior reward value can be set to 0.1, and the enhanced score enhanced_score = min(1.0, original_score + 0.1), ensuring that the enhanced score enhanced_score does not exceed the maximum value of 1.0, where original_score represents the first support score.

[0073] It should be noted that the prior reward value of 0.1 is only an example. In practical applications, it can be determined based on human experience or experimental optimization. For example, different candidate values ​​such as 0.05, 0.08, 0.1, 0.12, and 0.15 can be tried during the experiment, and the candidate value that makes the citation correction performance (such as using the F1 score to evaluate the citation correction performance) optimal can be selected as the prior reward value.

[0074] Of course, in addition to directly increasing the prior reward value, there are other ways to enhance the score, such as multiplying by a reward coefficient, which will not be listed in this application.

[0075] Step S204: Based on the enhanced score and the second support score corresponding to the first support score, select the valid citation documents corresponding to each fact text from the target document set, and generate the citation-corrected answer corresponding to the citation answer based on the valid citation documents.

[0076] Understandably, both the enhancement score and the second support score reflect the degree of consistency between the document content and the factual text. A higher score indicates that the document is a more valid source of citation for the factual text, meaning it is more likely to be a correct citation. Based on this, this application can select documents from the target document set that have the highest possible scores for the factual text, according to the enhancement score and the second support score corresponding to the first support score. These documents serve as valid citations for the factual text, and the cited documents are corrected accordingly to form a citation correction response.

[0077] One possible method for generating citation correction responses is as follows: First, based on the valid citation documents corresponding to each fact text, generate citation correction fact texts for each fact text. For example, one possible implementation is to obtain the identifier of the valid citation document corresponding to each fact text and insert it before the punctuation mark at the end of the corresponding fact text to obtain the citation correction fact texts for each fact text. For chart-type citations, a special format such as [[^id]] is used, and each citation occupies a separate line. Next, obtain the line number of each fact text in the citation-corrected response, concatenate the citation correction fact texts belonging to the same line together, and connect the citation correction fact texts belonging to different lines with line breaks to obtain the initial correction response. Finally, compress the multiple consecutive line breaks in the initial correction response to the target number to obtain the citation correction response.

[0078] Specifically, for the system, the factual text of each paragraph is on the same line. Therefore, this application only needs to record the line number of each factual text in the cited response, which is equivalent to recording its paragraph number. To place factual text belonging to the same paragraph on the same line, this application can group the cited correction factual text by line number, directly concatenate cited correction factual text with the same line number, and connect cited correction factual text with different line numbers using line breaks, thereby maintaining the original paragraph structure.

[0079] Considering that the initial corrected response may contain multiple consecutive line breaks, for aesthetic purposes, these consecutive line breaks can be compressed to the target number to obtain the citation corrected response. For example, three or more consecutive line breaks can be compressed into two.

[0080] Optionally, leading and trailing whitespace can be removed to improve the aesthetics of the citation correction response.

[0081] Optionally, in addition to outputting the citation correction response, you can also output a detailed scoring log for each fact text and global token usage statistics for the entire correction process.

[0082] Optionally, the detailed scoring log may include: the support score and enhancement score of the original cited document for each fact text, and whether the cited document was retained in the citation correction response; it may also include the support score of any newly inserted document for each fact text. Thus, the detailed scoring log can comprehensively display the correction content and reasons for the citation-based responses, making it convenient for users to view.

[0083] The post-citation correction method provided in this application obtains a set of citation-bearing answers and target documents generated by a target large language model. Considering that large language models typically annotate citations on a fact-by-fact basis when generating citation-bearing answers, this application first decomposes the citation-bearing answers into at least one fact text to ensure that the granularity of subsequent citation correction remains consistent with the original generation granularity. Next, a pre-trained target re-ranking model is used to determine the support score of each fact text and each document in the target document set, objectively quantifying the degree to which a document supports the fact text. Because this application leverages the deep semantic interaction capabilities of the re-ranking model to calculate a support score based on factual support, subsequent citation filtering using this support score as a selection criterion can effectively distinguish between documents that truly support the fact text and semantically similar but actually unsupportive difficult negative samples, thereby significantly improving the accuracy and reliability of citation filtering.

[0084] Considering that the existing factual and document citation relationships in the original cited responses often contain strong prior information—that is, the documents cited by the target large language model and the corresponding factual texts usually have a high correlation—this application uses the support scores of factual texts and documents with citation relationships in the cited responses as the first support score and the remaining support scores as the second support score. The first support score is then enhanced to be higher than the original first support score. By proactively improving the support scores of the documents cited by the target large language model, this application can effectively compensate for the potential underestimation of cited documents by the target re-ranking model. Finally, based on the enhanced score and the second support score corresponding to the first support score, valid citation documents corresponding to each factual text are selected from the target document set. Based on these valid citation documents, a citation correction response corresponding to the cited response is generated. This not only eliminates erroneous citation documents that cannot effectively support the factual text but also proactively matches and adds correct citation documents that were not originally cited but have supporting functions, significantly improving the completeness and accuracy of citation correction.

[0085] In some embodiments of this application, the process of “step S204, selecting effective citation documents corresponding to each fact text from the target document set based on the enhancement score corresponding to the first support score and the second support score” is described.

[0086] This embodiment can implement the filtering process through pre-configured thresholds. Considering the unequal task costs of filtering erroneous citation documents and supplementing missing citation documents, specifically, the high cost of erroneous citations requires a higher threshold, while the high cost of missing citations requires a lower threshold. To flexibly adapt to the correction strictness of different scenarios and achieve a better balance between citation accuracy and recall, this application provides a dual-threshold filtering mechanism. For citation filtering tasks, a preset filtering threshold can be used, and for citation insertion tasks, a preset insertion threshold can be used.

[0087] Based on this, this application can group the enhancement score and the second support score corresponding to the first support score according to the fact text, to obtain a score group corresponding to each fact text. Each score group corresponding to a fact text includes the second support score of that fact text and all documents in the target document set, or the score group includes the enhancement score of that fact text and cited documents in the target document set, and the second support score of that fact text and uncited documents. For ease of description below, each enhancement score and each second support score in the score group is used as a reference score.

[0088] Optionally, each group can be reordered in one go to obtain a complete score vector of dimension R, where R represents the number of documents in the target document set, that is, the number of reference scores in the group.

[0089] Therefore, for each factual text, the following processing is performed: For documents in the target document set that are not referenced by the fact text (i.e., documents in the target document set that do not have a reference relationship with the fact text), based on the score group corresponding to the fact text, documents with a reference score greater than the preset insertion threshold are selected from the unreferenced documents and used as the first candidate documents corresponding to the fact text. For documents in the target document set that have been referenced by factual text (i.e., documents in the target document set that have a reference relationship with factual text), then based on the grouping corresponding to the factual text, select documents from the referenced documents whose reference scores are greater than the preset filtering threshold, and use them as the second candidate documents corresponding to the factual text.

[0090] In other words, if the factual text does not cite any documents and it is necessary to supplement the factual text with missing citation documents, then the insertion threshold is used to filter the candidate documents that need to be inserted at the end of the factual text; if the factual text has cited documents and it is necessary to check for errors in the cited documents, then the filtering threshold is used to filter out citation documents with incorrect citations.

[0091] To explain in more detail, for each factual text: Regarding the documents cited by that factual text (which must exist in the target document set), if the reference score between the cited document and the factual text is greater than the filtering threshold, it means that the cited document can, to some extent, confirm, corroborate, or deduce the factual content expressed in the factual text (i.e., there is factual support between the factual text and the documents cited by it). In this case, the cited document can be considered a candidate document corresponding to the factual text (i.e., the second candidate document mentioned above). Conversely, if the reference score between the cited document and the factual text is less than or equal to the filtering threshold, it means that the cited document cannot confirm, corroborate, or deduce the factual content expressed in the factual text. If the factual content described by the factual text is not supported by facts, meaning there is a citation error in the cited document, then the cited document is directly filtered out to prevent the factual text from continuing to cite the cited document and causing citation errors. For documents that are not cited by the factual text but exist in the target document set, if the reference score between the document and the factual text is greater than the insertion threshold, it means that there is factual support between them, and the document can be used as a candidate document corresponding to the factual text (i.e., the first candidate document mentioned above). Conversely, if the reference score between the document and the factual text is less than or equal to the insertion threshold, then the document is filtered out.

[0092] In one possible implementation, for factual text that does not cite any documents in a cited response (hereinafter referred to as target factual text for ease of explanation), it can be first determined whether the character length of the target factual text is greater than a preset length threshold (e.g., 10 characters). If so, based on the grouping corresponding to the target factual text, the first and second candidate documents corresponding to the target factual text are filtered according to the above process, thereby filtering valid citation documents. For target factual texts with a negative result, the process of filtering valid citation documents corresponding to each factual text from the target document set based on the enhanced score and the second support score corresponding to the first support score is not performed. That is, when the character length of the target factual text is less than or equal to the length threshold, valid citation documents are not filtered for that target factual text. Since excessively short sentences lack sufficient information, citation supplementation can easily introduce noise. Therefore, this application only performs citation supplementation on long factual texts without cited documents.

[0093] Considering that relying solely on filtering thresholds and insertion thresholds for citation screening may retain too many citations, while a single factual text typically does not cite many citations, this application, after passing the threshold screening, can also select up to m candidate documents with the highest reference scores from the first and second candidate documents corresponding to each factual text as the valid citation documents corresponding to that factual text, where m is a preset positive integer, such as 3.

[0094] For example, suppose a factual text corresponds to candidate documents 1-10 (candidate documents 1-3 are the first candidate documents, and candidate documents 4-10 are the second candidate documents), and the reference scores between these 10 candidate documents and the factual text are 0.5, 0.4, 0.7, 0.5, 0.6, 0.9, 0.2, 0.4, 0.8, and 0.4, respectively. Then, if three candidate documents are selected, candidate document 3 (with a reference score of 0.7), candidate document 6 (with a reference score of 0.9), and candidate document 9 (with a reference score of 0.8) can be considered as valid citation documents for the factual text. If two candidate documents are selected, candidate document 6 and candidate document 9 can be considered as valid citation documents for the factual text. If one candidate document is selected, candidate document 6 can be considered as a valid citation document for the factual text.

[0095] This application achieves independent control over the deletion of erroneous citations and the supplementation of missing citations by distinguishing between cited and uncited documents and applying different thresholds for each. This allows for more flexible adaptation to the varying sensitivities of different business scenarios to the costs of incorrect and missing citations, improving citation accuracy while avoiding the excessive deletion of correctly cited documents.

[0096] In other embodiments of this application, the aforementioned filtering threshold and insertion threshold are not empirical constants, but are determined jointly by a systematic scan of the labeled dataset (i.e., optimization based on multiple candidate threshold scans) and a balance of multidimensional factors. Optionally, the multidimensional factors include: the relative magnitude of miscitation cost and missed citation cost (dimensional A), the asymmetry of filtering and insertion tasks (dimensional B), the coupling relationship of prior reward values ​​(dimensional C), the score distribution shift of the target re-ranking model (dimensional D), the corpus domain and document length (dimensional E), the original citation quality of the target large language model (dimensional F), and the retrieval recall scale R (dimensional G).

[0097] The process of determining the filtering threshold and the insertion threshold is explained in detail below to enable those skilled in the art to better understand.

[0098] The methods for determining (calibrating) the insertion threshold and the filtering threshold are similar, but the meanings of the ground truth labels used in the validation set differ. Specifically, each sample in the validation set includes a first fact text sample, a document sample, and ground truth labels. The ground truth labels corresponding to the insertion threshold indicate whether a document sample should be inserted for the first fact text sample, while the ground truth labels corresponding to the filtering threshold indicate whether the document sample truly supports the first fact text sample. The first fact text sample originates from real, cited response samples, and the document sample is a document from the target document set. The process for determining the target document set is the same as the process for determining the target document set described earlier; please refer to the previous description for details.

[0099] It should be noted that the number of samples in the above validation set can be adjusted according to the number of available first fact text samples and corresponding document samples, but preferably no less than 500 samples to ensure the statistical significance of the threshold calibration results. For example, in one embodiment, the validation set contains 1,000 to 5,000 samples.

[0100] It should also be noted that the true labels for each sample in the validation set were determined through manual annotation. Taking the filtering threshold as an example, at least two annotators with relevant domain knowledge can independently determine whether a document sample truly supports the first factual text sample; discrepancies are decided by a third expert. After annotation, at least 10% of the samples are randomly selected for independent review to ensure that the annotation error rate does not exceed 5%.

[0101] After obtaining the above validation set, this application can classify each sample in the validation set according to each candidate threshold in the preset candidate threshold set, obtain the predicted classification result corresponding to each sample, and generate the classification performance index corresponding to the candidate threshold according to the predicted classification result and the true label corresponding to all samples in the validation set.

[0102] Research has revealed the following typical patterns in threshold selection: when the candidate threshold τ is too low (e.g., <0.3), the recall rate approaches 1 but the precision drops rapidly, easily leading to bloat; when the candidate threshold τ is too high (e.g., >0.7), the precision increases but the recall rate drops significantly, resulting in a large number of missed citations; at the same time, when the candidate threshold τ is in the high-density transition region of re-ranked scores (usually [0.4, 0.5]), the F1 score reaches a local optimum. To obtain sufficient sampling granularity within the optimal range and comprehensively examine the impact of high and low thresholds on performance, this application can optionally construct a candidate threshold set within the range [0.05, 0.95] with a step size of 0.05, for example, {0.05, 0.10, 0.15, 0.20, 0.25, 0.30, 0.35, 0.40, 0.45, 0.50, 0.55, 0.60, 0.65, 0.70, 0.75, 0.80, 0.85, 0.90, 0.95}. This 0.05 step size ensures that at least three candidate thresholds of 0.40, 0.45, and 0.50 are included within the range [0.4, 0.5], thereby capturing fine-grained inflection points. Based on this, this application can use the default threshold of 0.45 as the out-of-the-box configuration for both the filtering and insertion thresholds when the two thresholds are not calibrated in a new scene or model version.

[0103] In a preferred implementation, the range and step size of the candidate threshold set can be adjusted according to the corpus domain and document length (dimension E). For example, for long documents or domains with dense multi-terms, the distribution of support scores and enhancement scores may be generally low. In this case, the lower limit of scanning can be extended to 0.00 to moderately reduce the target threshold and avoid recall rate collapse. For short text domains, the scanning range can be shrunk to [0.30, 0.80] and the encryption step size can be increased to 0.02.

[0104] Of course, the above process for determining the candidate threshold set is merely an example and is not intended to limit this application.

[0105] It should also be noted that the purpose of setting a candidate threshold set in this application is not to restrict the target threshold to be taken from the candidate threshold set, but to use it as a reference value for the specific value of the target threshold.

[0106] Taking the true label as 0 or 1 as an example, for each sample in the validation set, this application can use the target re-ranking model to calculate the support score between the first fact text sample and the document sample. Then, depending on whether the document sample is cited by the first fact text sample, it is decided whether to perform prior enhancement processing on the support score (as mentioned above, a prior reward value is introduced in the prior enhancement processing. The value of the prior reward value may be affected by the citation quality (dimension F) of the citation-based answer generated by the target large language model. The higher the citation accuracy, the larger the value of the prior reward value).

[0107] For each candidate threshold in the candidate threshold set, this application can compare the final score (support score or enhancement score) of each sample with the candidate threshold. If the final score is greater than the candidate threshold, the document sample is determined to be a predicted positive document; otherwise, the document sample is determined to be a predicted negative document. Thus, the prediction classification result corresponding to each sample is obtained.

[0108] For each sample in the validation set, if the document sample in the sample is a predicted positive document and the true label is 1, then the sample is labeled as a true positive (TP). If the document sample in the sample is a predicted positive document and the true label is 0, then the sample is labeled as a false positive (FP). If the document sample in the sample is a predicted negative document and the true label is 1, then the sample is labeled as a false negative (FN). If the document sample in the sample is a predicted negative document and the true label is 0, then the sample is labeled as a true negative (TN).

[0109] Based on the above markers, the classification performance metrics corresponding to each candidate threshold can be calculated, such as true positive rate (TPR), true negative rate (TNR), precision, recall, F1 score, and balanced accuracy (BAcc).

[0110] Then, based on the classification performance index corresponding to each candidate threshold in the candidate threshold set, the objective function value corresponding to each candidate threshold is determined.

[0111] Optionally, the objective function used to calculate the objective function value can be selected according to the application scenario. For example, for general question-answering scenarios, the objective function is the F1 score; for academic and financial reporting scenarios, where accuracy is emphasized, the objective function can be: ,in, This represents the objective function value corresponding to the candidate threshold τ. This represents the precision corresponding to the candidate threshold τ. This represents the recall rate corresponding to the candidate threshold τ. This indicates the preset minimum acceptable recall rate. This represents the preset penalty coefficient; for scenarios with imbalanced classes, the objective function takes the balance accuracy, etc.

[0112] Therefore, when generating classification performance indicators in the preceding text, the specific indicator type can be determined based on the query scenario corresponding to the cited answer. Thus, only the classification performance indicators used to calculate the objective function value corresponding to the scenario need to be calculated, instead of calculating all indicators.

[0113] Optionally, the specific form of the objective function and the hyperparameters within the function (such as...) , The cost of miscitation and omission can be determined by the relative magnitude of the costs of miscitation and omission (dimensional A) and the corpus domain and document length (dimensional E). For example, the higher the cost of miscitation, the more emphasis is placed on accuracy, while the cost can be appropriately reduced for long document domains. To avoid recall rate collapse.

[0114] Next, the target threshold is determined based on the objective function values ​​corresponding to all candidate thresholds in the candidate threshold set.

[0115] Optionally, the candidate threshold with the largest corresponding objective function value can be selected from the candidate threshold set as the base threshold, and the base threshold can be determined as the target threshold.

[0116] To avoid overfitting the candidate threshold to the validation set, this embodiment can take the base threshold and its numerically adjacent candidate thresholds based on the scanning results with a step size of 0.05 as described above. For example, if the base threshold is 0.50, then the adjacent candidate thresholds are 0.45 and 0.55. The target threshold can then be obtained by averaging the thresholds whose corresponding objective function values ​​are above a preset percentage (e.g., 95%) of the base function value among the adjacent candidate thresholds and the base threshold. Here, the base function value refers to the objective function value corresponding to the base threshold.

[0117] For example, assuming the base threshold is 0.50, the base function value is 0.90, and 95% of the base function value is 0.855; the two adjacent candidate thresholds are 0.45 and 0.55 respectively, the objective function value corresponding to the candidate threshold of 0.45 is 0.87, and the objective function value corresponding to the candidate threshold of 0.55 is 0.84 (less than 0.855), then the target threshold is the average of 0.50 and 0.45, which is 0.475.

[0118] Optionally, after obtaining the filtering threshold and insertion threshold through the above calibration process, the threshold can be adjusted before or during use by one or more of the following correction constraint rules.

[0119] Rule 1: Adjust the filtering threshold based on the relative magnitude of the cost of miscitation and the cost of missing citation (dimensional A). If the cost of miscitation is high, increase the filtering threshold; if the cost of missing citation is high, decrease the filtering threshold. Rule 2: Based on the asymmetric constraint between filtering and insertion (dimension B), the insertion threshold must be greater than or equal to the filtering threshold. For example, if the calibration results do not meet this rule, the insertion threshold will be increased to the calibrated filtering threshold to ensure that the insertion operation is more conservative than the filtering operation. Rule 3: Based on the coupling relationship of prior reward values ​​(dimension C), when the prior reward value increases, the filtering threshold is adjusted downward by an equal amount; when the prior reward value decreases, the filtering threshold is adjusted upward by an equal amount. Rule 4: Based on the score distribution shift (dimension D) of the target re-ranking model, after the model version is upgraded, fine-tuned, or the domain or language is changed, the previous calibration process should be automatically re-executed, and the old threshold should not be directly used. Rule 5: Dynamically adjust the insertion threshold based on the retrieval recall size R (dimension G). The larger R is, the larger the insertion threshold becomes, in order to suppress the risk of false positives.

[0120] It should be noted that the above five rules are just examples. In addition, there are other ways to adjust the filtering threshold and the insertion threshold. For example, the number of validation set samples accumulated in the online data logs can be counted. When the number of samples reaches the threshold (such as 10,000), and after it has been calibrated at least once through the calibration process mentioned above, the filtering threshold and the insertion threshold can be recalibrated.

[0121] In summary, this embodiment provides a systematic, data-driven dual-threshold calibration method. It scans candidate thresholds on the validation set with fine-grained precision (e.g., a step size of 0.05), optimizes the result using a scenario-adaptive objective function, and finally ensures robustness through neighborhood smoothing. This dual-threshold calibration method not only considers the inherent differences between filtering and insertion tasks but also takes into account multiple dimensions such as the cost of miscitation and missed citations, model score distribution, and corpus domain characteristics. This ensures that the final determined dual thresholds achieve a better overall balance between precision and recall in the citation correction system, effectively combining out-of-the-box dual thresholds with scenario-adaptive optimization.

[0122] In some other embodiments of this application, the process of “step S202, determining the support score of each fact text and each document in the target document set through a pre-trained target reordering model” is described.

[0123] Optionally, the process of calculating the support score may include: inputting each fact text and each document into the target reordering model to obtain the log probabilities of the first target token and the second target token generated by the model, wherein the first target token indicates that the document supports the fact text and the second target token indicates that the document does not support the fact text, and calculating the support score based on the log probabilities of the first target token and the second target token.

[0124] To explain in more detail, the core task of the target re-ranking model is to determine whether a given document can factually support the factual text. In essence, this core task is a binary classification problem, with the classification results including: the document supports the factual text and the document does not support the factual text.

[0125] Traditional discriminative re-ranking models typically output a binary classification probability by adding a classification head to the output layer. However, this approach requires modifications to the model architecture and is relatively complex in terms of training and inference. In contrast, this application employs a generative re-ranking model to accomplish the aforementioned binary classification discriminative task.

[0126] It is understandable that generative models have powerful semantic understanding and text generation capabilities. By designing appropriate prompts, the model can be guided to directly output a specific token representing the judgment result (such as the first target token "yes" or the second target token "no").

[0127] Based on this, the present application can construct the following prompt instructions: "System: Judge whether the Document meets the requirements based on the Query and the Instruct provided in the context. Note that the answer can only be "yes" or "no". User: <instruct>: {instruction text} <query>: {fact text} <context>{No cited answer} <document>: {Reference document content}.

[0128] The aforementioned reference document content refers to the content of the document from which it needs to be determined whether it supports the factual text. Optionally, this reference document content may include the following structured fields: document source, document type, document title, document description, and document body. By describing the reference document content with structured fields, richer contextual information can be provided, making it easier for the model to understand and obtain more accurate judgment results.

[0129] Optionally, to force the target reordering model to skip the internal thought chain and directly output "yes" or "no", a specific suffix string can be added to the end of the prompt, such as "<|im_start|>assistant\n <think> \n\n< / think> When the model sees this specific suffix string, it considers the thought process to be complete, and can generate only its own judgment result "yes" or "no", thus achieving efficient scoring for single token generation.

[0130] After inputting the aforementioned prompts into the target reordering model, the model, after forward computation, will generate a logits vector of length equal to the vocabulary size at the output layer. The logits values ​​corresponding to the first target token "yes" and the second target token "no" are extracted from this vector and denoted as follows: , Take the exponent of the two logarithmic probabilities respectively to restore them to their corresponding probabilities. , Then normalize the two.

[0131] Optionally, the support score can be calculated using the following formula (1).

[0132] Formula (1); in, This represents the support score between the i-th fact text and the j-th document. This represents the normalized probability of the first target term predicted by the target reordering model for the i-th factual text and the j-th document. Let e ​​represent the normalized probability of the second target term predicted by the target reordering model for the i-th factual text and the j-th document, where e is the natural constant and serves as the base of the exponential function, and log represents taking the logarithm.

[0133] Therefore, it can be seen that the process of calculating the support score in this embodiment does not require modification of the model architecture. It only needs to construct the core task of the target re-ranking model as a binary classification problem of the generative re-ranking model. It can give full play to the advantages of the generative re-ranking model in deep semantic understanding, output intuitive and easy-to-understand single-token judgment results, and has stronger interpretability. Compared with keyword matching or traditional discriminative re-ranking models, it can more accurately identify those semantically similar but actually unsupported factual texts, making the judgment results more accurate and reliable, and thus the support score more accurate and reliable.

[0134] In some further embodiments of this application, the training process of the aforementioned target reordering model is described in detail.

[0135] First, training data is acquired, which includes second-fact text samples and corresponding positive example documents and multiple negative example documents. Positive example documents are those that genuinely support the second-fact text samples, while negative example documents are those that do not. In one possible implementation, each training data point is uniformly organized into a sample structure of "one second-fact text sample (as a query), a non-citation answer (as the text context), one or more positive example documents, and several negative example documents." To construct a sufficient quantity of high-quality training data with varying difficulty levels, this embodiment provides the following two complementary data construction strategies, which can be used selectively or in combination in practical applications.

[0136] The first strategy is a data construction approach based on citation annotation reuse. This strategy reuses existing citation-annotated attribution corpora, directly converting manually annotated citation relationships into positive example supervision signals, and mining difficult negative examples within the same candidate document set. Specifically, it obtains citation-annotated attribution corpora, with each corpus sample including a query, a citation-annotated response sample generated for that query, and the corresponding candidate document set; the following processing is performed on each corpus sample: First, using the same citation parser as described above for decomposing factual texts, the citation-annotated response samples are broken down into multiple second-fact text samples sentence by sentence. All citation tags annotated in each second-fact text sample are parsed and recorded, ensuring consistency between the decomposition granularity during training and inference phases. Second, citation tags are removed from the entire citation-annotated response samples to obtain citation-free responses, serving as a shared discourse context for all second-fact text samples. This allows the model to learn supporting relationship judgments within a discourse context consistent with the inference phase. Third, for each citation-annotated second-fact text sample, the documents it cites in the candidate document set are used as positive example documents (when a second-fact text sample is annotated with citations, the document it references in the candidate document set is used as a positive example document). When the annotation of a second factual text sample references multiple documents, each referenced document is treated as a positive example document. Documents in the candidate document set that are not referenced by the second factual text sample are treated as difficult negative example documents. These difficult negative example documents belong to the same candidate document set retrieved for the same query as the positive example documents. They are similar in topic to the second factual text sample but do not constitute factual support. They are the objects that the target re-ranking model needs to distinguish. When the number of difficult negative example documents in the same candidate document set is insufficient, several documents are randomly selected from the candidate documents of other corpus samples as simple negative example documents to supplement them. These simple negative example documents are unrelated to the topic of the second factual text sample and are easy to distinguish.

[0137] It should be noted that when constructing the above training data, all citation tags were removed from the second fact text samples used as queries and the non-citation answers used as context, to avoid citation tags leaking supervision labels and to prevent the model from relying on citation tags rather than semantic content for judgment. Therefore, this strategy can construct a difficulty gradient containing both difficult and easy negative examples at no cost based on manual annotation, enabling the model to learn fine-grained discrimination boundaries from topic relevance to precise support. The first data construction strategy described above can be represented by the following pseudocode: "for corpus samples in citation-tagged attribution corpus: # corpus samples = {query, citation-tagged response samples, candidate document set}, Context = clear citation tags (citation-tagged response samples)" For the second fact text sample, break it down sentence by sentence (with quoted response sample): Citation tag set = Parsed citation tags (second fact text sample) if the referenced identifier set is empty: continue The set of positive examples = (candidate document set, set of referenced identifiers) retrieved by identifier. Difficult negative example set = Unreferenced documents in the candidate document set Simple negative example set = randomly selected from candidate documents of other corpus samples (padded to a preset number). Query = Remove citation tags (second fact text sample) for example documents in collection of example documents: `yield (Query, Context, positive example documents, set of difficult negative examples ∪ set of simple negative examples)`.

[0138] The second approach is a data construction strategy based on the RAG system. This strategy is geared towards real-world RAG scenarios and generates positive and negative examples by evaluating the support level of each retrieved document using a decision model. Specifically, it obtains citation-based response samples, which can be derived from publicly available citation attribution / fact-checking datasets, online request-response logs accumulated by the target large language model in actual deployment, or optionally, generated in real-time by the target large language model for the query. The citation-based response samples are then decomposed into multiple second-fact text samples, and a set of retrieved documents corresponding to each second-fact text sample is determined. For each second-fact text sample and each document in its retrieved document set, the decision model evaluates the support level of the document for that second-fact text sample. Based on a preset support level threshold, documents with a support level not lower than the threshold are classified as positive examples, and documents with a support level lower than the threshold are classified as negative examples. The decision model can be any of the following: Firstly, a second language model with a larger parameter scale. Optionally, the second language model first decomposes the answer sample into multiple sub-claims, and then judges the factual consistency and logical consistency of each sub-claim with respect to the reference document, and gives the degree of support accordingly. Since the second language model has a larger parameter scale and more accurate judgment, it is equivalent to distilling its support judgment capability into a lighter target reordering model, thereby improving the computational efficiency of support score in the deployment phase while ensuring the quality of judgment.

[0139] Secondly, the target re-ranking model itself uses the yes / no support scores mentioned earlier as the degree of support. When using this method, quality cleaning can be performed on positive example documents: if the support scores of all positive example documents labeled with a certain second fact text sample are lower than a preset positive example threshold, then the second fact text sample is removed to suppress noise annotation; and among the non-positive example documents, several documents are selected first from low to high support scores as difficult negative example documents, and randomly supplemented if insufficient. The above second data construction strategy can be represented by the following pseudocode: for citation-based response samples in (Citation Attribution / Fact Check Dataset ∪ Online Request-Response Log): # Optional: Can also be generated in real-time by the target large language model For the second fact text sample in the decomposition (with cited response sample): for document in the set of documents corresponding to the sample answer: Support Level = Decision Model (Second Fact Text Sample, Document) # Decision Model = Larger Second Language Model or Target Reordering Model Itself If the support level is greater than or equal to the support level threshold: Positive example pair .append((second fact text sample, document)) else: Negative example pair .append((second fact text sample, document)) Of the two data construction strategies described above, the first focuses on reusing manual citation annotations and generating negative samples with controllable granularity and difficulty levels, while the second focuses on simulating noise distribution in real RAG scenarios and using a judgment model to mine difficult negative examples and clean low-quality samples. These two strategies complement each other. Optionally, for chart-type citations, samples containing chart description documents can be constructed separately to enhance the model's ability to support and discriminate chart citations. Each second factual text sample corresponds to one positive example document and several (e.g., three) negative example documents, forming a set of comparative learning samples.

[0140] After obtaining the training data as described above, the support scores of the second fact text samples in the training data with positive and negative documents can be determined by the pre-built initial re-ranking model. Then, based on the support scores of the second fact text samples with positive and negative documents, a contrastive learning loss is generated, and the initial re-ranking model is trained based on the contrastive learning loss to obtain the target re-ranking model.

[0141] Optionally, the InfoNCE (information noise-contrastive estimation) contrastive learning loss function can be used, with the loss function taking the form: Formula (2); in, Indicates the contrast learning loss. This represents a second factual text sample. This represents a positive example document. Indicates a negative example document. The support score between the second fact text sample and the positive example document is indicated (the calculation process can be referred to the formula (1) above). The support score represents the second fact text sample and the negative example document. Indicates the number of negative example documents. This represents the temperature parameter.

[0142] Therefore, this embodiment uses the calculated contrastive learning loss as the optimization objective and updates the parameters of the initial re-ranking model using the backpropagation algorithm. In actual training, efficient training frameworks and strategies can be employed. For example, the scalable lightweight infrastructure for fine-tuning (SWIFT) training framework (ms-swift) can be used, with multiple GPUs (e.g., 8) configured for parallel training. The DeepSpeedZeRO-3 optimization strategy (the third stage of the DeepSpeed ​​zero-redundancy optimizer) can be used for full parameter fine-tuning, and an appropriate learning rate (e.g., 6e-6) and number of training epochs (e.g., 5 epochs) can be set. After training, the target re-ranking model can be obtained.

[0143] As described above, this application provides an end-to-end fine-tunable training pipeline. By designing two complementary training data generation strategies—similarity-driven and RAG system-driven—it can effectively construct diverse, high-quality training data. The similarity-driven strategy generates negative samples with controllable granularity and graded difficulty, helping the model learn the fine boundary from semantic similarity to precise support. The RAG system-driven strategy simulates noise distribution in real-world scenarios, improving the model's robustness in practical applications. Furthermore, by combining InfoNCE contrastive learning loss, the target re-ranking model can output a higher "yes" probability for positive examples and a higher "no" probability for negative examples, thereby widening the gap between the log probabilities of "yes" and "no." This allows for more accurate support relationship judgments when scoring new factual texts and documents, thus improving the accuracy of subsequent post-citation correction.

[0144] To further verify the post-citation correction effect of the target re-ranking model based on the above pre-trained model, this application provides the following metrics, such as citation recall, citation precision, citation F1 score, and balanced accuracy (BAcc), to comprehensively measure the citation correction performance.

[0145] Citation recall is used to measure the proportion of missed citations in the system, that is, how many documents that should have been cited were successfully inserted into the actual text. The calculation formula is: Formula (3); in, Let S represent the citation recall rate, where S represents the set of all factual texts in the cited responses to be corrected, and s is a single factual text in S. c represents the complete set of citations (corrected) annotated in the factual text s, where c is... One of the cited documents, This represents a natural language reasoning function that determines whether a cited document c can factually support s. The implied relation label indicates that c supports the facts in s. It indicates that it exists.

[0146] The citation recall rate mentioned above represents the proportion of factual texts in which at least one citation supports the factual text. The lower the proportion, the more serious the citation omissions.

[0147] Citation precision is used to measure the proportion of miscitations in a system, that is, how many of the citations annotated by the system are truly correct. The formula is: Formula (4); in, Indicates citation precision. The set of redundant citations (i.e., invalid citations) in factual text s, that is, citations that satisfy one of the following two conditions. The set of: (Condition 1) Even if c is taken from If removed, the remaining set of citations can still support the factual text s (condition 2).

[0148] The citation precision rate mentioned above represents the proportion of truly valid citations among all punctuated citations. The lower the proportion, the more serious the miscitation.

[0149] The citation F1 score is the harmonic mean of citation recall and citation precision, used to comprehensively measure the citation correction performance of the system. The calculation formula is: Formula (5); in, The F1 score represents the citation score, which avoids the biased evaluation caused by a single metric (such as pursuing only recall at the expense of precision, or vice versa), and is a more comprehensive performance metric.

[0150] Balanced accuracy is used to handle imbalanced positive and negative samples. For example, when most factual texts contain correct citations, relying solely on accuracy may mask the model's inadequacy in identifying negative examples (i.e., incorrect citations). The formula for balancing accuracy is: Formula (6); Here, BAcc represents the balanced accuracy, TPR represents the true positive rate, and TNR represents the true negative rate. This balanced accuracy, by averaging the performance of the two classes, provides a fairer evaluation of the model's performance on the minority class.

[0151] Because there may be inconsistencies in sentence-level classification between the citation correction results and the standard answer, directly calculating the overall index will lead to bias. Therefore, this application provides a sentence-level evaluation method based on character overlap.

[0152] Specifically, the system first segments the citation-corrected answers predicted by the system and the standard answers annotated by humans. Then, the sentences in the two answers are matched and aligned based on character overlap (for example, the similarity between the sentences in the two answers is calculated using the Longest Common Subsequence (LCS), and alignment is achieved based on the similarity). Finally, based on the completed sentence alignment, the citation metrics (such as the citation precision and citation recall mentioned above) of each aligned sentence pair are calculated sentence by sentence, and the citation metrics of all sentences are summarized or averaged to obtain the final evaluation result.

[0153] The sentence-level evaluation method based on character overlap provided in this embodiment can effectively overcome the evaluation noise caused by differences in sentence boundary division, making the evaluation results more accurate and reliable.

[0154] In some other embodiments of this application, a post-citation processing correction system is provided, such as... Figure 3 The post-citation correction system provided in this application includes a four-layer architecture: RAG system layer 301, post-citation correction layer 302, model training pipeline layer 303, and service deployment layer 304.

[0155] In one possible physical deployment, the four-layer architecture of this post-citation correction system can be deployed as follows: Figure 1 The terminal and / or server shown.

[0156] The RAG system layer 301 is responsible for receiving user queries, retrieving relevant document sets from the knowledge base through a retrieval tool, and generating citation-included answers from the target large language model. The citation post-processing correction layer 302 is responsible for performing citation post-processing correction on the citation-included answers. The correction process is as described in the previous embodiments and will not be repeated here. The model training pipeline layer 303 is responsible for building training data, fine-tuning the target re-ranking model based on the training data, and evaluating model performance based on preset metrics. The service deployment layer 304 is responsible for carrying the trained model parameters and providing efficient GPU inference services. It provides a support score calculation interface to the citation post-processing correction layer and receives the target re-ranking model produced by the model training pipeline layer for deployment and hot updates.

[0157] The following section provides a detailed introduction to the specific architecture of the service deployment layer and its production-grade reliability assurance mechanisms.

[0158] To meet the high concurrency and low latency requirements of the online citation correction service, this application designs and implements a dual-service microarchitecture. The two services are deployed in a containerized manner through the application container engine (Docker) and communicate using the Docker Bridge network.

[0159] Specifically, the first service is the citation filtering API service (port 8000). This citation filtering API service is built on FastAPI and Gunicorn and provides representational state transfer (REST) ​​API interfaces, such as the filter_citations interface and the calculate_metrics interface. The filter_citations interface is used to receive citation-corrected responses and return citation-corrected responses, while the calculate_metrics interface is used to evaluate the correction effect online.

[0160] The citation filtering API service implements semaphore concurrency control; for example, the default maximum concurrent requests are set to 10 to prevent service overload caused by sudden traffic spikes. It also supports request timeout control (optional, default timeout is 300 seconds) and an exponential backoff retry mechanism (optional, maximum retries of 3) to improve service robustness. Furthermore, the citation filtering API service provides optional data logging functionality, asynchronously writing online requests (such as citation-supported responses) and response data (such as citation-corrected responses) to a log file in line-oriented JSON (JavaScript Object Notation) format for subsequent offline analysis and model iteration.

[0161] The second service is the reordering inference service (port 8001), which is built on FastAPI and the vLLM GPU inference engine and is responsible for loading pre-trained target reordering models. The vLLM inference engine supports TensorParallel multi-GPU inference, which can distribute the computational load of the target reordering model across multiple GPUs, thereby reducing the latency of individual requests.

[0162] The reranking inference service also provides the `rerank_with_custom_instruction` API interface, which receives fact text and a set of target documents and returns batch support scores. Furthermore, the reranking inference service can enable prefix caching to accelerate inference for repetitive or similar queries.

[0163] In addition, this application integrates a multi-layered reliability assurance mechanism on the basis of the dual-service microarchitecture.

[0164] In one possible implementation, deep liveness checks can be implemented. Specifically, conventional health checks can only verify whether the service process is alive, and cannot resolve the issue where a vLLM internal worker process crashes but the Hypertext Transfer Protocol (HTTP) layer still returns a 200 error. To address this issue, this application can actually execute a lightweight inference request (e.g., calling the reordering interface with a pre-defined simple factual text and document) to verify the end-to-end functional availability of the target reordering model. If this lightweight inference request fails, the service is determined to be in an unhealthy state.

[0165] In another possible implementation, hot model reloading can be implemented. Specifically, when the deep liveness check detects that the target reordered model is unavailable, the system can automatically trigger a model reloading process. This process includes: first, performing GPU memory cleanup (calling torch.cuda.empty_cache()) to release the occupied GPU memory; then, destroying the distributed parallel state (calling destroy_model_parallel()) to reset the model parallel environment; and finally, reloading the model parameters, thereby attempting to resume the reordered inference service without restarting the entire container.

[0166] In another possible implementation, the circuit breaker pattern can be implemented. Specifically, to prevent the target reordering model from entering an infinite reload loop due to underlying errors, the system can maintain a continuous reload failure counter. Each time a hot reload fails, the counter is incremented by 1. When the number of consecutive reload failures exceeds a preset threshold (e.g., 3 times), the circuit breaker trips, and the system actively exits the current process (calling sys.exit(1)). At this time, the entire container is automatically restarted by the Docker container's restart strategy (e.g., restart:unless-stopped), avoiding an infinite reload loop.

[0167] In another possible implementation, garbage collection (GC) optimization can be implemented. Specifically, referring to the memory management strategy of vLLM, long-lived objects (such as model parameter caches, fixed hint templates, etc.) can be frozen using gc.freeze(), excluding them from garbage collection scans to reduce the impact of garbage collection on inference latency.

[0168] It should be noted that the above dual-service microarchitecture and reliability guarantee mechanism are merely examples and are not intended to limit this application.

[0169] The following examples demonstrate the practical effectiveness of the post-citation correction method in the financial investment research scenario.

[0170] Input a Chinese financial market analysis text generated by the target large language model, containing 12 citations. The corresponding target document set consists of 12 relevant listed company financial reports and industry research reports. Configure the following parameters: set the filtering threshold to 0.45, set the insertion threshold to 0.45, and enable the combined filtering and insertion mode.

[0171] The system first uses a citation parser to parse the Chinese financial market analysis text into eight independent factual sentences (i.e., factual text), extracting 21 citation tags from them. Then, for each factual sentence, a reordering inference service is invoked to calculate a support score between that sentence and all 12 documents, resulting in an 8×12 support score matrix. Based on the enhanced scores and a dual-threshold decision mechanism, the system ultimately filters out three low-relevance erroneous citations and automatically matches and adds appropriate citation tags to two factual sentences that originally lacked citation tags.

[0172] After being corrected by the system described in this application, the citation accuracy of the financial research report improved from the original 68.3% to 83.1%. This example demonstrates that the method described in this application has significant practical value in the financial field, which is characterized by dense technical terminology and extremely high accuracy requirements.

[0173] The above describes a post-citation correction method provided by embodiments of this application. The following describes the apparatus for performing the above post-citation correction method.

[0174] Please see Figure 4 , Figure 4 This is a schematic diagram of a post-citation processing correction device provided in an embodiment of this application. Figure 4 As shown, the post-citation processing correction device may include: The fact text decomposition unit 401 is used to obtain the cited answer and target document set generated by the target large language model, decompose the cited answer into at least one fact text, and the target document set includes all documents cited in the cited answer; The support score calculation unit 402 is used to determine the support score of each fact text and each document in the target document set through a pre-trained target re-ranking model. The support score represents the degree to which a document supports the fact text. The prior enhancement unit 403 is used to take the support scores of factual texts and documents with citation relationships in the cited answer as the first support score, take the remaining support scores as the second support scores, perform prior enhancement processing on the first support score, and obtain the enhanced score corresponding to the first support score. The enhanced score is higher than the corresponding first support score. The citation correction unit 404 is used to filter the valid citation documents corresponding to each fact text from the target document set according to the enhancement score and the second support score corresponding to the first support score, and generate a citation correction answer with the citation answer according to the valid citation documents.

[0175] Each module in the aforementioned post-citation processing and correction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0176] This application also provides an electronic device, which may include at least one processor and a memory connected to the processor, wherein: Memory is used to store computer programs; The processor is used to execute computer programs to enable the electronic device to implement any of the post-citation correction methods provided in the embodiments of this application.

[0177] refer to Figure 5 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0178] like Figure 5 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. When the electronic device is powered on, the RAM 603 also stores various programs and data required for the operation of the electronic device. The processing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0179] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, memory cards, hard drives, etc.; and communication devices 609. Communication device 609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0180] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the post-citation correction methods provided in this application.

[0181] This application also provides a computer-readable storage medium carrying one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the post-citation correction methods provided in this application.

[0182] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0183] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0184] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0185] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).< / document> < / context> < / query> < / instruct>

Claims

1. A post-citation processing correction method, characterized in that, include: Obtain the cited answer and target document set generated by the target large language model, decompose the cited answer into at least one fact text, and the target document set includes all documents cited in the cited answer; A pre-trained target re-ranking model is used to determine the support score between each fact text and each document in the target document set, whereby the support score characterizes the degree to which a document supports the fact text. The support score of the factual text and document with citation relationship in the cited answer is taken as the first support score, and the remaining support scores are taken as the second support score. The first support score is subjected to a priori enhancement processing to obtain the enhanced score corresponding to the first support score. The enhanced score is higher than the corresponding first support score. Based on the enhanced score corresponding to the first support score and the second support score, valid citation documents corresponding to each factual text are selected from the target document set, and a citation-corrected answer corresponding to the citation answer is generated based on the valid citation documents.

2. The post-citation processing correction method according to claim 1, characterized in that, The step of filtering valid citation documents corresponding to each factual text from the target document set based on the enhancement score corresponding to the first support score and the second support score includes: The enhancement score corresponding to the first support score and the second support score are grouped according to the fact text to obtain a group corresponding to each fact text, wherein each enhancement score and each second support score in the group are used as reference scores; For each of the aforementioned fact texts, process them separately: For documents in the target document set that are not referenced by the fact text, based on the grouping corresponding to the fact text, documents with a reference score greater than a preset insertion threshold are selected from the unreferenced documents and used as the first candidate documents corresponding to the fact text. For the documents in the target document set whose factual text has been referenced, based on the grouping corresponding to the factual text, documents with a reference score greater than a preset filtering threshold are selected from the referenced documents and used as the second candidate documents corresponding to the factual text. From the first and second candidate documents corresponding to each fact text, select at most m candidate documents with the highest reference scores as the valid citation documents corresponding to each fact text, where m is a positive integer.

3. The post-citation processing correction method according to claim 2, characterized in that, Before filtering the valid citation documents corresponding to each factual text from the target document set based on the enhanced score corresponding to the first support score and the second support score, the method further includes: For target fact text that does not cite any documents, determine whether the character length of the target fact text is greater than a preset length threshold. If the determination result is negative, do not perform the step of filtering valid citation documents corresponding to each fact text from the target document set based on the enhanced score corresponding to the first support score and the second support score.

4. The post-citation processing correction method according to claim 2 or 3, characterized in that, Using the insertion threshold and the filtering threshold as target thresholds respectively, the process of determining the target thresholds includes: Obtain a validation set, wherein each sample in the validation set includes a first factual text sample, a document sample, and a real label. The real label corresponding to the insertion threshold is used to indicate whether the document sample should be inserted for the first factual text sample. The real label corresponding to the filtering threshold is used to indicate whether the document sample truly supports the first factual text sample. For each candidate threshold in the preset candidate threshold set, each sample in the validation set is classified according to the candidate threshold to obtain the predicted classification result corresponding to each sample. Based on the predicted classification results corresponding to all samples in the validation set and the true label, the classification performance index corresponding to the candidate threshold is generated. Based on the classification performance index corresponding to each candidate threshold, the objective function value corresponding to each candidate threshold is determined, and based on the objective function values ​​corresponding to all candidate thresholds in the candidate threshold set, the target threshold is determined.

5. The post-citation processing correction method according to claim 4, characterized in that, Determining the target threshold based on the objective function values ​​corresponding to all candidate thresholds in the candidate threshold set includes: From the candidate threshold set, the candidate threshold with the largest corresponding objective function value is selected as the basic threshold, wherein the objective function value corresponding to the basic threshold is the basic function value; From the set of candidate thresholds, determine the candidate thresholds that are numerically adjacent to the base threshold; The target threshold is obtained by averaging the thresholds whose corresponding objective function values ​​are above a preset percentage of the basic function value among the adjacent candidate thresholds and the basic threshold.

6. The post-citation processing correction method according to claim 4, characterized in that, The type of the classification performance index is determined based on the query scenario corresponding to the cited answer.

7. The post-citation processing correction method according to claim 1, characterized in that, The step of determining the support score of each fact text relative to each document in the target document set using a pre-trained target re-ranking model includes: Each fact text and each document are input into the target reordering model to obtain the log probabilities of the first target token and the second target token generated by the model. The first target token indicates that the document supports the fact text, and the second target token indicates that the document does not support the fact text. The support score is calculated based on the logarithmic probabilities of the first target token and the second target token, respectively.

8. The post-citation processing correction method according to claim 1, characterized in that, The step of performing prior augmentation processing on the first support score to obtain the augmentation score corresponding to the first support score includes: A preset prior reward value is added to the first support score to obtain the enhanced score corresponding to the first support score.

9. The post-citation processing correction method according to claim 1, characterized in that, The training process of the target reordering model includes: Acquire training data, which includes a second factual text sample and corresponding positive example documents and multiple negative example documents. The positive example documents are those that actually support the second factual text sample, and the negative example documents are those that do not support the second factual text sample. The support scores of the second factual text sample with the positive example document and each of the negative example documents are determined by a pre-built initial re-ranking model. Based on the support scores of the second factual text sample and the positive and negative example documents respectively, a contrastive learning loss is generated; The initial re-ranking model is trained using the contrastive learning loss to obtain the target re-ranking model.

10. The post-citation processing correction method according to claim 1, characterized in that, The step of decomposing the cited answer into at least one factual text includes: The citation markers in the citation-containing response are identified and removed using a preset regular expression to obtain a citation-free response. The uncited answer was segmented into sentences to obtain multiple sentence texts; Sentences that meet preset conditions from the plurality of sentence texts are removed, and each remaining sentence text is treated as a fact text to obtain at least one fact text. The preset conditions include at least one of the following: the sentence text is a title sentence, the sentence text is a quotation sentence in chart format or other preset format, or the sentence text is a Chinese numbered title.

11. The post-citation processing correction method according to claim 1, characterized in that, The step of generating the citation-corrected answer corresponding to the citation-included answer based on the valid citation document includes: Based on the valid citation document corresponding to each fact text, generate a citation-corrected fact text corresponding to each fact text; Obtain the line number of each fact text in the cited response, concatenate the citation correction fact texts belonging to the same line in all the citation correction fact texts together, and connect the citation correction fact texts belonging to different lines with a newline character to obtain the initial correction response; The initial corrected response is obtained by compressing multiple consecutive newline characters into the target number.

12. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the post-citation correction method as described in any one of claims 1 to 11.

13. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the post-citation correction method as described in any one of claims 1 to 11.

14. A computer storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the post-citation correction method as described in any one of claims 1 to 11.

Images