A method and system for preventing tampering and verifying the effect level aging and law citation

Abstract

The application discloses a validity level time limit solution and law reference anti-tampering verification method and system, and belongs to the field of legal information processing. The application introduces conflict resolution driven by validity level and effective time limit in mixed retrieval and reordering, eliminates low-level or invalid conflict evidence according to the principle that higher-level law is superior to lower-level law and new law is superior to old law, and retains a high-authority evidence set; generates an answer draft with reference marks based on a constrained prompt; performs semantic consistency verification and character-level word-by-word verification, and still performs word-by-word comparison of law article numbers and clause contents on a statement determined to be implied by semantic verification, so as to intercept references whose numbers or expressions are tampered with although they imply semantics; and performs confidence calibration on generated content, triggers a refusal or manual review when the content is unreliable, significantly improves the reference number accuracy, traceability and time limit correctness of a legal question and answer system, effectively suppresses field-specific hallucinations, and realizes active safety interception in high-risk legal scenarios.

Description

Technical Field

[0001] This invention belongs to the field of legal information processing, and in particular relates to a method and system for verifying the resolution of validity hierarchy and statute of limitations and preventing tampering of legal citations. Background Technology

[0002] Applying large language models to question-and-answer scenarios such as legal consultation, legal provision retrieval, and case-assisted analysis has become a technological trend. To mitigate the illusion of large language models, existing technologies have proposed various retrieval enhancement and generation verification schemes.

[0003] For example, patent application CN122364448A discloses a method and system for assisting in drafting judicial documents based on retrieval enhancement, including: structuring and segmenting case facts and constructing a legal provision database; generating reasoning-oriented judicial summaries; hierarchical query rewriting to generate retrieval expressions; forming multi-granularity query expressions from coarse to fine; vectorized legal provision retrieval and reordering; performing multi-level parallel retrieval in a pre-constructed legal provision vector retrieval database, and weighting and sorting according to semantic similarity and the legal provision's validity level to output a set of relevant legal provisions; constructing structured input generation; segmented and controlled generation of judicial reasoning text; and adopting a phased generation strategy to sequentially complete the determination of the case's nature, analysis of the points of contention, legal application argumentation, and expression of the judgment conclusion, ultimately piecing together a complete reasoning text. Although this method considers the legal provisions' validity level, it remains at a static weighting level; furthermore, it lacks any verification process for the reasoning text, making it difficult to detect character-level modifications and affecting the accuracy of the legal provision index.

[0004] Furthermore, directly applying existing search enhancement technologies to legal Q&A still suffers from the following technical shortcomings determined by the objective attributes of legal texts and which are difficult to resolve with existing verification methods: First, semantic verification and entity matching verification struggle to detect word-for-word modifications to legal provision numbers or key wording. Consistency verification based on natural language reasoning models determines the semantic implication, contradiction, or neutrality between the generated statement and the evidence; fact verification based on entity identification and entity matching thresholds determines whether key entities appear and whether the matching degree meets the standard. Neither approach can effectively intercept errors like "the generated statement is semantically fully implied by the evidence, and the key entity is present, but the cited legal provision number has been changed from 'Article 38' to 'Article 39'." This is because such modifications remain semantically valid and the entities still match, but legally constitute a critical error that is difficult for non-professionals to identify. Legal texts are highly sensitive to the word-for-word accuracy of provision numbers and wording; such word-for-word modifications must be reliably detected through character-level word-for-word comparison.

[0005] Secondly, existing search and verification methods fail to resolve conflicts in the hierarchy and statute of limitations of legal evidence. Legal texts exhibit differences in their hierarchy of validity (the hierarchy among constitutions, laws, administrative regulations, local regulations, departmental rules, and judicial interpretations) and evolve over time (new and old laws, effective and repealed laws, and laws before and after revisions). The same legal issue may have conflicting provisions in texts of different hierarchical levels or at different times of implementation. Existing search rankings, based solely on semantic relevance or entity association, may present invalid or lower-level clauses as evidence to the model, inducing a domain-specific illusion of responding based on invalid or lower-level clauses; existing consistency verification also fails to determine whether cited clauses are still currently valid.

[0006] Third, existing verification methods often employ a regeneration strategy when verification fails, lacking quantitative measurement and automatic interception capabilities for unreliable output. When verification failure only triggers regeneration, if the evidence itself is insufficient or conflicting, regeneration may still repeatedly generate unreliable content and ultimately present it to the user, failing to proactively refuse to answer or transfer the issue to human intervention when it is unreliable. Summary of the Invention

[0007] To overcome the technical shortcomings of legal question-and-answer systems, such as low accuracy, poor traceability, and insufficient timeliness due to the use of low-level or expired regulations during the retrieval stage, modification of legal provision numbers or key wording during the generation stage (leading to the inability of natural language reasoning models to recognize them), and the lack of proactive interception mechanisms for unreliable output, this invention provides a method and system for resolving validity hierarchy and timeliness issues and verifying legal provision citations to prevent tampering. This is achieved by introducing evidence conflict resolution driven by validity hierarchy and timeliness during the hybrid retrieval and reordering stages; employing a dual verification process combining semantic consistency verification and character-level word-by-word verification during the generation verification stage; and performing a word-by-word comparison between the legal provision number and the content of statements determined by semantic verification to be implicit, thereby intercepting semantically implicit citations where the number or wording has been modified; and calibrating the confidence level of the generated content, triggering rejection or manual review when unreliable, significantly improving the accuracy, traceability, and timeliness of legal provision citation numbers.

[0008] Based on the above technical objectives, this embodiment provides a method for resolving the validity hierarchy and time-limited validity and verifying the anti-tampering of legal citations, including the following steps: Step 1: Obtain the user's legal question and answer query, and perform intent classification, intent rewriting and question decomposition in sequence to obtain the standardized search query and its intent tag. Based on the standardized search query, retrieve candidate evidence fragments from the legal corpus. Each candidate evidence fragment is associated with legal attribute information including at least the legal rank and effective date. Step 2: Sort the candidate evidence fragments by relevance, and resolve conflicts between conflicting candidate evidence fragments based on their validity level and effective date, in order to determine the set of evidence used to generate the answer; Step 3: Using standardized retrieval queries, intent tags, and evidence sets as constraints, guide the large language model to perform constrained decoding based on the evidence set to generate a draft answer, and label each statement in the draft answer with the cited evidence fragments; Step 4: Perform dual verification on the draft answer, including semantic consistency verification and character-level word-by-word verification. Calculate the confidence level of the draft answer based on the results of the dual verification. Based on the relationship between the confidence level and a preset threshold, decide whether to output the draft answer, trigger rejection, or transfer to human intervention.

[0009] This invention addresses the domain-specific illusion of generating incorrect answers based on low-level or expired provisions by introducing the rank of validity and effective date as conflict resolution criteria during the retrieval process. Furthermore, a dual verification architecture combining semantic consistency verification and character-by-character verification forces a character-by-character comparison of legal provision numbers and clause content based on original text offsets after semantic matching of statements. This precisely intercepts number modifications or key wording changes that natural language reasoning models cannot detect, while simultaneously checking the current validity of cited clauses. Finally, based on multi-dimensional signals, confidence fusion and tiered threshold handling proactively trigger rejection or referral to human intervention when global confidence is insufficient or core citations are abnormal, forming a complete risk closed loop and significantly improving the accuracy, traceability, and timeliness of citation numbers in the legal question-and-answer system.

[0010] Preferably, step 1 specifically includes: The intent recognition model is invoked to classify the intent of legal question and answer queries, and the original legal question and answer queries are normalized and decomposed into sub-questions based on the sequence-to-sequence rewriting model, outputting one or more normalized search queries and their intent tags; Based on the legal corpus, dense and sparse searches are performed in parallel on the standardized search query to recall candidate evidence fragments. The fragments are then merged and normalized according to their inverse ranking to remove duplicates, thus obtaining the candidate evidence fragments.

[0011] Furthermore, dense retrieval is based on a sentence vector dual-tower encoder; sparse retrieval is based on a probabilistic relevance retrieval model.

[0012] Preferably, the relevance ranking of the candidate evidence fragments in step 2 includes: calling an interactive re-ranking model to perform legal question-and-answer queries and score the relevance of the candidate evidence fragments, and then arranging them in descending order.

[0013] Preferably, the conflict resolution in step 2 includes: In accordance with the order of the Constitution, laws, administrative regulations, local regulations and departmental rules, and judicial interpretations, or the rule that higher-level laws prevail over lower-level laws, special laws under the same level prevail over general laws, and new laws prevail over old laws, candidate evidence fragments that present conflicting clauses are selected and discarded. Evidence fragments with higher legal force or later effective dates are retained. Conflicting fragments that are eliminated are not included in the candidate evidence fragment set but are retained in the evidence chain by reducing their weight or marking the conflict relationship.

[0014] The system clearly outlines progressive rules and a step-by-step adjudication strategy, which fully aligns with the legally established principles for resolving conflicts of validity. This ensures that evidence selection is legally irrefutable and preserves the complete chain of evidence to meet judicial auditing requirements.

[0015] Preferably, step 3 specifically includes: organizing the standardized search query, intent tag, and evidence set into constraints according to a preset prompt template, calling the large language model for constrained decoding, requiring the generation of a draft answer based only on the given evidence set, and forcibly labeling the evidence fragment number on which each generated statement is based, thereby obtaining the evidence fragment cited by each statement.

[0016] Preferably, the semantic consistency verification in step 4 includes: segmenting the draft answer into sentences, forming a premise hypothesis pair between each generated sentence and the evidence fragment it cites, calling a natural language reasoning model to determine whether the relationship between the two is implied, contradictory, or neutral, and marking sentences that are determined to be contradictory or neutral as unfounded or conflicting statements. Character-level word-by-word verification is performed only on statements that have been determined to be implied through semantic verification. It is used to compare the legal provision number and / or clause content in the statement with the original text of the cited evidence word by word, and to perform statute of limitations verification on the cited evidence to determine whether the cited evidence is currently valid at the time of verification. The statute of limitations verification combines the effective date, repeal or revision status of the cited evidence fragment with the time of verification, and adds a statute of limitations doubt mark to non-currently valid clauses that have not yet taken effect, have been repealed or revised.

[0017] The natural language reasoning model is refined to determine the three categories of logic (implication, contradiction, and neutrality), and character-level verification is specified to be performed only on implied sentences, avoiding the performance burden of full comparison; specific anomaly labeling types are given (untraceable, mismatched numbering, modified content, and questionable timeliness), making the verification results quantifiable and classifiable; different labels correspond to different handling strategies, and this fine-grained classification significantly enhances the interpretability and handling flexibility of the system.

[0018] More preferably, the character-level word-by-word verification includes: locating the original text in the source document according to the character start and end offset of the evidence fragment, calculating the character-level alignment and overlap between the cited fragment and the original text, marking it as untraceable when the overlap is lower than a set threshold, and marking it as a mismatch of number or content modification when the legal provision number or clause content is inconsistent with the original text.

[0019] Furthermore, the character-level alignment overlap is a weighted sum of the proportion of the longest common substring and the normalized similarity based on edit distance; wherein the set threshold is 0.85.

[0020] Furthermore, the word-by-word comparison of legal provision numbers involves standardizing and unifying the legal provision numbers in terms of Arabic numerals and Chinese numerals, full-width and half-width characters, and the writing format of clauses and items before verification, and using whether the words are consistent after standardization as the criterion for judgment.

[0021] Furthermore, the word-by-word comparison includes a character-by-character check of the legal provision number, article, clause, item level identifiers, and key wording that indicates legal consequences. The key wording includes "above" and "below" indicating the scope, "shall" and "may" indicating the nature of the obligation, and expressions of scope, period, and amount.

[0022] Preferably, the calculation of the confidence level of the draft answer based on the results of the double verification in step 4 includes: Temperature scaling calibration is applied to the output probabilities of each sentence generated by the large language model, and multiple forward samplings are performed on the generation and natural language inference judgment through Monte Carlo random inactivation. By combining the implied probability of semantic verification, character-level alignment overlap, and calibrated generation probability, the comprehensive confidence score of each generated statement and the global confidence score of the entire draft answer are calculated. Based on the comparison results of the comprehensive confidence score and the global confidence score with the preset first threshold and the preset second threshold, respectively, and whether there are any mismatched numbers, content modifications, or timeliness doubts, the system decides to remove or highlight the corresponding statements. When the global confidence score is lower than the preset second threshold or there are core legal provisions cited by the marked statements, the system will trigger a refusal to answer or transfer to human verification. Otherwise, the system will output a draft answer with a verification mark.

[0023] Furthermore, the comprehensive confidence score is a weighted sum of the implied probability, character-level alignment overlap, post-calibration generation probability, and sampling consistency. The global confidence score is a weighted average of the overall confidence scores of each sentence, calculated by sentence length. The first threshold is preset to 0.6, and the second threshold is preset to 0.5.

[0024] Preferably, the process includes an offline phase before retrieval: segmenting the legal corpus into evidence fragments according to natural clauses or semantic paragraphs, and establishing metadata records for each evidence fragment; The metadata record includes at least the unique identifier of the source document, the level of validity, the effective date, the repeal or revision status, the article number, and the start and end offsets of the characters. The level of validity, the effective date, and the repeal or revision status are automatically obtained by connecting to an authoritative legal database or by parsing legal texts using rules, and are managed in a versioned manner and maintained synchronously with the update of legal events.

[0025] On the other hand, the present invention also provides a system for verifying the resolution of validity hierarchy and the prevention of tampering in legal citation, used to implement the aforementioned method for verifying the resolution of validity hierarchy and the prevention of tampering in legal citation, including: The query rewriting module is used to classify intents and decompose questions based on users' legal Q&A queries, resulting in standardized search queries and intent tags; The hybrid retrieval module is used to sequentially classify intent, rewrite intent, and decompose questions to obtain standardized retrieval queries and their intent tags. Based on the standardized retrieval queries, candidate evidence fragments are retrieved from the legal corpus. Each candidate evidence fragment is associated with legal attribute information including at least the legal rank and the effective date. The reordering and validity conflict resolution module is used to sort the candidate evidence fragments by relevance and resolve conflicts of candidate evidence fragments based on validity rank and effective date, so as to determine the evidence set used to generate the answer. The constrained generation module is used to guide the large language model to perform constrained decoding based on the evidence set, using standardized retrieval query, intent tag and evidence set as constraints, and to annotate the evidence fragments cited in each statement in the answer draft; The dual verification module is used to perform dual verification on the draft answer, including semantic consistency verification and character-level word-by-word verification; The confidence calibration and decision output module is used to calculate the confidence level of the draft answer based on the results of the double verification, and decide whether to output the draft answer, trigger rejection, or transfer to human intervention based on the relationship between the confidence level and a preset threshold.

[0026] Compared with the prior art, the present invention has the following beneficial effects: (1) By linking semantic consistency verification and character-level word-by-word verification, the legal provision number and clause content are still forcibly compared word-by-word for statements that are semantically implied. This can effectively intercept deep illusion errors that are semantically implied by evidence, where key entities are present, but the number or wording has been modified. Such errors cannot be identified by natural language reasoning models and can only be discovered through precise character-level matching, thereby greatly improving the accuracy of citation numbers and the traceability of the original text.

[0027] (2) In the evidence reordering stage, conflict resolution rules driven by the order of validity and the effective date are introduced, prioritizing the retention of evidence fragments with higher validity and later effective dates, and eliminating conflict provisions that have been repealed, expired, or of lower order. At the same time, the current validity of the cited clauses is checked in the verification stage, blocking errors in answers based on expired or lower order provisions from both the pre- and post-generation stages, significantly improving the timeliness and legal compliance of the answers.

[0028] (3) Based on multidimensional signals, calculate the comprehensive confidence and global confidence. When the confidence is lower than the threshold or there are abnormal markers in the core legal provisions, actively trigger the refusal to answer or transfer to manual verification. This is different from the scheme of simply regenerating after verification failure. It establishes a safe closed loop in high-risk legal scenarios where it is better not to answer than to answer incorrectly. Attached Figure Description

[0029] Figure 1 This is a flowchart illustrating the method for resolving validity hierarchy and verifying the anti-tampering of legal citations provided in this embodiment of the invention.

[0030] Figure 2 This is a schematic diagram of the hybrid retrieval, validity conflict resolution, and reordering process provided in the embodiment.

[0031] Figure 3 This is a schematic diagram of the double verification and confidence calibration process.

[0032] Figure 4 This is a schematic diagram of the structural operation logic of the validity hierarchy time-limited resolution and legal provision citation anti-tampering verification system provided for the embodiment. Detailed Implementation

[0033] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the embodiments described below are intended to facilitate the understanding of the present invention and do not constitute any limitation thereof.

[0034] The technical concept of this invention is as follows: Addressing the technical deficiencies in legal question-and-answer systems, such as low accuracy, poor traceability, and insufficient timeliness due to the use of low-level or expired regulations during the retrieval stage, modifications to legal provision numbers or key wording during the generation stage that prevent natural language reasoning models from recognizing them, and the lack of proactive interception mechanisms for unreliable output, this invention provides a method and system for resolving validity order and timeliness issues and verifying legal provision citations to prevent tampering. This is achieved by introducing evidence conflict resolution driven by validity order and timeliness in the mixed retrieval and reordering stages; employing a dual verification process combining semantic consistency verification and character-level word-by-word verification during the generation verification stage; and performing word-by-word comparison between legal provision numbers and clause content for statements determined to be implicit through semantic verification, thereby intercepting semantically implicit citations where the number or wording has been modified; and calibrating the confidence level of the generated content, triggering rejection or manual review when unreliable, significantly improving the accuracy, traceability, and timeliness of legal provision citation numbers.

[0035] like Figure 1 As shown, this embodiment is deployed on a server cluster with graphics processor acceleration capabilities, and includes offline and online phases. The specific implementation steps are as follows: In the offline phase, the legal corpus is preprocessed. The legal corpus includes currently effective laws, administrative regulations, local regulations, departmental rules, judicial interpretations, and judgments. Each article and document is segmented into evidence fragments of 128 to 256 Chinese characters in length, with an overlap of 32 characters between adjacent fragments, based on natural clauses or semantic paragraphs. Metadata records are established for each fragment, including the unique identifier of the source document, document title, legal effect level, effective date and repeal or revision status, article number, and the start and end character offsets of the fragment within the full text of the source document. The legal attribute metadata, such as legal effect level, effective date, repeal or revision status, is automatically obtained by connecting to authoritative legal databases or by parsing legal texts using rules to obtain information on publication, implementation, repeal, and revision. This metadata is managed in a versioned manner and maintained synchronously with legal updates to ensure the accuracy of legal effect level and timeliness determination. Subsequently, a dual index was constructed: the dense vector index adopts a sentence vector dual-tower structure (12 layers of self-attention encoding layers, 768 hidden dimensions, 12 attention heads, and 768-dimensional sentence vectors obtained by mean pooling and L2 normalization), and is written into a hierarchical navigable small-world index (maximum number of connections per node is 32, and the search width during construction is 200); the sparse inverted index is built after segmenting Chinese fragments and identifying legal proper nouns, and is used for retrieval by the probabilistic relevance retrieval model (word frequency saturation coefficient is 1.2, and length normalization coefficient is 0.75).

[0036] The online phase will proceed as follows: Step 1: Obtain the user's legal question and answer query, and perform intent classification, intent rewriting and question decomposition in sequence to obtain a standardized search query and its intent tags. Based on the standardized search query, retrieve candidate evidence fragments from the legal corpus. Each candidate evidence fragment is associated with legal attribute information including at least the legal rank and effective date.

[0037] In this embodiment, the intent recognition model uses a bidirectional encoder connected to a normalized exponential function classification header to classify queries into one of the following: legal provision query, case query, concept explanation, or procedure consultation. The query rewriting model uses a sequence-to-sequence structure to normalize and decompose colloquial queries containing multiple sub-questions into independent sub-queries.

[0038] like Figure 2 As shown, the normalized retrieval query is executed in parallel with dense retrieval based on sentence vector dual-tower encoder and sparse retrieval based on probability relevance retrieval model. Sparse retrieval is used to accurately hit sparse legal identifiers that are difficult to hit by semantic vectors, such as legal provision numbers. Each retrieves a set number of candidate fragments, and merges them by normalization and deduplication according to the inverse ranking to obtain a set of candidate evidence fragments. Each candidate evidence fragment is associated with its source document identifier, provision number, validity level, effective date, and the start and end character offsets in the source document.

[0039] In this embodiment, dense retrieval recalls 50 fragments with a search width of 128, and sparse retrieval recalls 50 fragments using a probabilistic relevance retrieval model. After deduplication using inverse ranking fusion (the fusion score is the sum of 1 / (60 + ranking number) of each path), the top 80 candidate fragments are retained. The source document identifier, clause number, validity level, effective date, and character offset of each fragment are read out together. When multiple subqueries are output, recall and resolution are performed on each subquery separately. The evidence sets obtained from each subquery are then provided to the constrained prompt generation module after deduplication and merging of evidence sources. The draft answer is organized into segments according to sub-questions.

[0040] Step 2: Sort the candidate evidence fragments by relevance, and resolve conflicts between conflicting candidate evidence fragments based on their validity level and effective date, in order to determine the set of evidence used to generate the answer.

[0041] In the embodiments, such as Figure 2 As shown, the interactive reordering model based on cross-encoder is called to score the query-fragment relevance of candidate evidence fragments and sort them in descending order. The cross-encoder reordering model (12 layers of self-attention encoding layers, which take the classification label, query, separator label, fragment, and separator label as input in sequence, and take the classification label bit through the linear layer and the logistic function to output the relevance score from 0 to 1) scores 80 candidates in descending order.

[0042] Subsequently, the conflict between the order of effectiveness and the statute of limitations is resolved: Candidate segments mapped to the same intent tag or determined by semantic clustering to address the same legal issue in pairs constitute a pair of premises. If the natural language reasoning model determines these segments to be contradictory, or if the rule template detects opposing provisions (e.g., one party is an obligatory provision while the other is a prohibitive provision, or different amounts or ranges are stipulated for the same matter), they are considered conflicting provisions. For candidate segments identified as conflicting, they are compared according to their order of effectiveness (Constitution > Law > Administrative Regulations > Local Regulations and Departmental Rules > Judicial Interpretations). If the orders are the same, special laws prevail over general laws, and new laws prevail over old laws. Segments with higher orders or later effective dates are retained, while eliminated segments are weighted and marked with conflict relationships before being retained in the evidence chain. Finally, the first eight segments are selected as the evidence set. For example, if both an old law provision and its revised new provision are recalled for the same issue, the new provision is retained in the generation process, while the old provision is retained in the evidence chain for future reference.

[0043] Step 3: Generate a draft answer based on the evidence set, and label each statement in the draft answer with the cited evidence fragments. Organize the standardized search query, intent tags, and evidence set into a constrained generation input according to a preset prompt template. Call the large language model for constrained decoding, requiring the generation of a draft answer based only on the given evidence. Forcibly label each generated statement with the evidence fragment number it is based on, and output the draft answer with sentence-level citation tags.

[0044] In this embodiment, the constrained generated prompt template is: [You are a legal information retrieval assistant, and you can only answer based on the following numbered evidence. You may not use information other than the evidence. Each sentence ends with the 'evidence number' to indicate the basis. Evidence: [1]...[2]...Question:...]. The base large language model is called, with a decoding temperature of 0.3 and a kernel sampling probability of 0.9, and the draft answer with sentence-level "evidence number" is output.

[0045] Step 4: Perform dual verification on the draft answer, including semantic consistency verification and character-level word-by-word verification. Calculate the confidence level of the draft answer based on the results of the dual verification. Based on the relationship between the confidence level and a preset threshold, decide whether to output the draft answer, trigger rejection, or transfer to human intervention.

[0046] Semantic consistency verification: The draft answer is segmented into sentences. For each generated sentence, a premise hypothesis pair is formed with the evidence fragments it cites. The natural language reasoning model is called to determine whether the relationship between the two is implied, contradictory, or neutral. Sentences that are determined to be contradictory or neutral are marked as unfounded or conflicting statements.

[0047] Character-level word-by-word verification and timeliness verification: For generated statements determined to be implicit, the original text of the cited evidence is located in the corresponding source document according to the character start and end offsets. The character-level alignment and overlap between the cited fragment in the generated statement and the original text of the source document are calculated, and the legal provision number and clause content in the cited statement are compared word-by-word. When the character-level alignment and overlap is lower than the set threshold, it is marked as untraceable. When the legal provision number or clause content is inconsistent with the original text of the source document, it is still marked as number mismatch or content modification even if the generated statement is determined to be implicit. In combination with the effective date, repeal or revision status of the cited evidence fragment and the verification time, it is determined whether the cited clause is currently valid at the verification time. For cited clauses that have not yet taken effect, have been repealed, or have been revised and are not currently valid, a timeliness doubt mark is added. The verification time is the current date by default.

[0048] Confidence calibration: Temperature scaling calibration is applied to the output probabilities of each sentence generated by the large language model. Multiple forward sampling is performed on the generation and natural language inference judgment through Monte Carlo random deactivation. The comprehensive confidence score of each generated sentence and the global confidence score of the entire draft answer are calculated by combining the implication probability, character-level alignment overlap, etc.

[0049] Tiered threshold handling: Statements judged as contradictory, or marked as having mismatched numbers, modified content, untraceable origin, or questionable timeliness, or statements with an overall confidence level below the first threshold, will be removed or highlighted as a warning; when the overall confidence level is below the second threshold, or when there is a core legal provision cited by the above-marked statement, given the high risk of legal consultation, a refusal to answer or a transfer to manual review process will be triggered, accompanied by a search of evidence chain, and a final legal judgment will be made manually; otherwise, a draft answer with verification citation marks, confidence score, and evidence chain will be output.

[0050] Specifically, such as Figure 3 As shown in the embodiment, the draft answer is segmented by periods and semicolons; for each generated sentence and its cited evidence, the cited evidence is taken as a premise and the generated sentence is taken as an assumption. The input is a natural language reasoning model fine-tuned by a pre-trained language model, and the output is three types of probabilities: implication, contradiction, and neutral. If the implication probability is >0.7, it is marked as implication; if the contradiction probability is >0.5, it is marked as contradiction; otherwise, it is marked as neutral. Contradictory or neutral sentences are marked as sentences without evidence.

[0051] For generated sentences deemed to contain implicit meaning, the legal provision number and directly quoted clause text are first extracted from the sentence using rules or models as cited fragments. Alternatively, during constrained generation, the large language model is required to mark the directly quoted content with delimiters to determine the cited fragments. Then, the cited original text is located in the source document according to the character offset of the fragment metadata. The proportion of the longest common substring between the cited text and the original text and the normalized similarity based on edit distance are calculated, and a weighted character-level alignment overlap is obtained. If the similarity is less than 0.85, it is marked as untraceable. Before checking character by character, the legal provision number is standardized and unified, unifying the different writing forms of Arabic numerals and Chinese numerals, full-width and half-width characters, and clause items into a standard form to avoid misjudging different writing forms of the same number as number mismatch. The standardized legal provision number (including the article, clause, and item level identifiers) is checked character by character against the clause content. If the number or key wording is inconsistent with the original text, even if the sentence is deemed to contain implicit meaning, it is still marked as number mismatch or content modification. For example, if the original evidence states "punishment for three years or less," but the generated sentence states "punishment for three years or more," although it is judged as implied due to semantic similarity, a word-by-word comparison still reveals that "less than" has been modified to "more than," marking the content modification. Further, considering the effective date, repeal or revision status of the cited passage, and the verification time (defaulting to the current date), if the cited clause was not in effect, had been repealed, or had been revised and is not currently valid at that time, a "statute of limitations in doubt" mark is added.

[0052] Temperature scaling calibration is applied to the output probabilities of generated sentence terms (the calibration temperature is fitted on the validation set to minimize the expected calibration error; in this embodiment, it is set to 1.4). Random deactivation of the generation and natural language inference models is enabled, performing 20 forward sampling iterations to statistically analyze self-consistency and decision stability. To reduce computational overhead, the Monte Carlo random deactivation sampling can be triggered only for statements with initially low overall confidence or statements containing core legal citations; the number of sampling iterations is configurable, and multiple sampling iterations of generation and natural language inference can be performed in parallel or in batches. The overall confidence is a weighted sum of implied probability, character-level alignment overlap, post-calibration generation probability, and sampling consistency, with weights of 0.35, 0.30, 0.20, and 0.15, respectively. The global confidence is a weighted average of the overall confidence of each sentence by sentence length. The weights, calibration temperature, alignment overlap threshold of 0.85, and the first and second thresholds are all preset values ​​and can be optimized on the validation set to minimize the expected calibration error and maximize the modification interception rate and rejection quality, respectively. The values ​​provided in this embodiment are just examples.

[0053] Sentences deemed contradictory, marked with mismatched numbers, modified content, untraceable origins, questionable timeliness, or with a comprehensive confidence level below the first threshold of 0.6 are removed or highlighted with a warning. When the global confidence level falls below the second threshold of 0.5, or when there are core legal citations that are affected (i.e., legal citations used as the basis for conclusions or when the intent tag is a legal citation query), the system does not directly answer, triggering a rejection and outputting "[The current question lacks sufficient and reliable legal basis; manual review is recommended]", along with a chain of evidence. Upon successful verification, a draft answer with citation tags, sentence-by-sentence confidence levels, and a chain of evidence is output. Furthermore, when a rejection is triggered due to insufficient evidence, an expanded search instruction is sent back to the retrieval stage, broadening the search width and recall quantity, and re-executing constrained generation, forming a maximum of two rounds of feedback loops. The first threshold, the second threshold, and the rejection trigger conditions can be configured according to the application scenario's requirements for rejection rate and accuracy.

[0054] A second aspect of the present invention provides a system for implementing the above-described method. The above scheme, without changing the parameters of the base large language model, resolves evidence conflicts through validity hierarchy and timeliness, intercepts character-by-character modifications by linking semantic verification and character-level word-by-word verification, and intercepts unreliable outputs through confidence calibration and rejection, such as... Figure 4 As shown, each module can be carried out by processing units such as retrieval, generation, verification, and adjudication, and coordinated through a message bus, including: The query rewriting module is used to classify intents and decompose questions based on users' legal Q&A queries, resulting in standardized search queries and intent tags; The hybrid retrieval module is used to sequentially classify intent, rewrite intent, and decompose questions to obtain standardized retrieval queries and their intent tags. Based on the standardized retrieval queries, candidate evidence fragments are retrieved from the legal corpus. Each candidate evidence fragment is associated with legal attribute information including at least the legal rank and the effective date. The reordering and validity conflict resolution module is used to sort the candidate evidence fragments by relevance and resolve conflicts of candidate evidence fragments based on validity rank and effective date, so as to determine the evidence set used to generate the answer. The constrained generation module is used to guide the large language model to perform constrained decoding based on the evidence set, using standardized retrieval query, intent tag and evidence set as constraints, and to annotate the evidence fragments cited in each statement in the answer draft; The dual verification module is used to perform dual verification on the draft answer, including semantic consistency verification and character-level word-by-word verification; The confidence calibration and decision output module is used to calculate the confidence level of the draft answer based on the results of the double verification, and decide whether to output the draft answer, trigger rejection, or transfer to human intervention based on the relationship between the confidence level and a preset threshold.

[0055] At the device level, the electronic device includes a memory and a processor. The processor is a heterogeneous combination of a graphics processing unit (GPU) and a central processing unit (CPU). The memory stores dual indexes, model weights, and computer programs. Dense retrieval and re-ranking, natural language reasoning verification, and generation / decoding are loaded onto different GPUs to achieve pipelined parallelism. Model training: The sentence vector dual-tower model is fine-tuned on legal question-and-answer / article pairing data using contrastive learning (in-batch negative sampling and difficult negative example mining, multiple negative sample ranking loss); the cross-encoder is fine-tuned using query-fragment relevance binary classification; the natural language reasoning model is pre-trained on a general natural language reasoning dataset and then fine-tuned on legal domain sentence-evidence three-classification labeled data; the intent recognition and query rewriting models are fine-tuned using labeled data and parallel corpora, respectively. All outputs of this invention are positioned as draft answers and chains of evidence with verification citations and confidence levels. The system does not make final judgments on legal issues; the final legal judgment and adoption right are reserved manually.

[0056] In summary, the method and system provided by this invention employ a dual verification process that combines semantic consistency verification with character-level word-by-word verification. For statements deemed implicit by semantic verification, a word-by-word comparison of the legal provision number and clause content is still performed. This effectively intercepts a unique type of error: "semantically implied by evidence, with key entities present, but the cited legal provision number or clause wording has been modified word-by-word." This type of error is semantically valid and matches at the entity level, and can only be detected through character-level word-by-word comparison, thus significantly improving the accuracy of legal provision numbers and the traceability of citations. Furthermore, the invention introduces conflict resolution between the order of validity and the effective date of evidence reordering, and performs timeliness verification during the verification process, suppressing the domain-specific illusion of answering based on expired or low-ranking clauses. Finally, the invention performs confidence calibration on the generated content and triggers rejection or manual processing when unreliable, unlike schemes that simply regenerate after verification failure, thus achieving automatic interception of unreliable output.

[0057] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for resolving the validity hierarchy and time-limited limitation and verifying the anti-tampering of legal citations, characterized in that, Includes the following steps: Step 1: Obtain the user's legal question and answer query, and perform intent classification, intent rewriting and question decomposition in sequence to obtain the standardized search query and its intent tag. Based on the standardized search query, retrieve candidate evidence fragments from the legal corpus. Each candidate evidence fragment is associated with legal attribute information including at least the legal rank and effective date. Step 2: Sort the candidate evidence fragments by relevance, and resolve conflicts between conflicting candidate evidence fragments based on their validity level and effective date, in order to determine the set of evidence used to generate the answer; Step 3: Using standardized retrieval queries, intent tags, and evidence sets as constraints, guide the large language model to perform constrained decoding based on the evidence set to generate a draft answer, and label each statement in the draft answer with the cited evidence fragments; Step 4: Perform dual verification on the draft answer, including semantic consistency verification and character-level word-by-word verification. Calculate the confidence level of the draft answer based on the results of the dual verification. Based on the relationship between the confidence level and a preset threshold, decide whether to output the draft answer, trigger rejection, or transfer to human intervention.

2. The method for resolving validity hierarchy and verifying legal citation tampering according to claim 1, characterized in that, Step 1 specifically includes: The intent recognition model is invoked to classify the intent of legal question and answer queries, and the original legal question and answer queries are normalized and decomposed into sub-questions based on the sequence-to-sequence rewriting model, outputting one or more normalized search queries and their intent tags; Based on the legal corpus, dense and sparse searches are performed in parallel on the standardized search query to recall candidate evidence fragments. The fragments are then merged and normalized according to their inverse ranking to remove duplicates, thus obtaining the candidate evidence fragments.

3. The method for resolving validity hierarchy and verifying legal citation tampering according to claim 1, characterized in that, Step 2, which involves ranking the candidate evidence fragments based on their relevance, includes: calling an interactive reordering model to perform legal question-and-answer queries and score the relevance of the candidate evidence fragments, and then arranging them in descending order.

4. The method for resolving validity hierarchy and verifying legal citation tampering according to claim 1, characterized in that, Conflict resolution in step 2 includes: In accordance with the order of the Constitution, laws, administrative regulations, local regulations and departmental rules, and judicial interpretations, or the rule that higher-level laws prevail over lower-level laws, special laws under the same level prevail over general laws, and new laws prevail over old laws, candidate evidence fragments that present conflicting clauses are selected and discarded. Evidence fragments with higher legal force or later effective dates are retained. Conflicting fragments that are eliminated are not included in the candidate evidence fragment set but are retained in the evidence chain by reducing their weight or marking the conflict relationship.

5. The method for resolving validity hierarchy and verifying legal citation tampering according to claim 1, characterized in that, The semantic consistency verification in step 4 includes: segmenting the draft answer into sentences, forming a premise hypothesis pair between each generated statement and the evidence fragment it cites, calling a natural language reasoning model to determine whether the relationship between the two is implied, contradictory, or neutral, and marking statements that are determined to be contradictory or neutral as unfounded or conflicting statements. Character-level word-by-word verification is performed only on statements that have been determined to be implied through semantic verification. It is used to compare the legal provision number and / or clause content in the statement with the original text of the cited evidence word by word, and to perform statute of limitations verification on the cited evidence to determine whether the cited evidence is currently valid at the time of verification. The statute of limitations verification combines the effective date, repeal or revision status of the cited evidence fragment with the time of verification, and adds a statute of limitations doubt mark to non-currently valid clauses that have not yet taken effect, have been repealed or revised.

6. The method for resolving validity hierarchy and verifying legal citation tampering according to claim 5, characterized in that, The character-level word-by-word verification includes: locating the original text in the source document according to the character start and end offset of the evidence fragment, calculating the character-level alignment and overlap between the cited fragment and the original text, marking it as untraceable when the overlap is lower than a set threshold, and marking it as mismatched number or tampered content when the legal provision number or clause content is inconsistent with the original text.

7. The method for resolving validity hierarchy and verifying legal citation tampering according to claim 1, characterized in that, Character-level alignment overlap is a weighted sum of the proportion of the longest common substring and the normalized similarity based on edit distance; Before verification, the legal provision numbers are standardized and unified in terms of Arabic numerals and Chinese numerals, full-width and half-width characters, and the writing style of clauses and items. The standard for judgment is whether the unified numbers are consistent character by character. The word-by-word comparison includes a character-by-character check of the legal provision number, article, clause, item level identifiers, and key wording that indicates legal consequences. The key wording includes words indicating the scope of the obligation such as "above" and "below", words indicating the nature of the obligation such as "shall" and "may", and expressions of scope, period, and amount.

8. The method for resolving validity hierarchy and verifying legal citation tampering according to claim 1, characterized in that, Step 4, which involves calculating the confidence level of the draft answer based on the results of the double verification, includes: Temperature scaling calibration is applied to the output probabilities of each sentence generated by the large language model, and multiple forward samplings are performed on the generation and natural language inference judgment through Monte Carlo random inactivation. By combining the implied probability of semantic verification, character-level alignment overlap, and calibrated generation probability, the comprehensive confidence score of each generated statement and the global confidence score of the entire draft answer are calculated. Based on the comparison results of the comprehensive confidence score and the global confidence score with the preset first threshold and the preset second threshold, respectively, and whether there are any mismatched numbers, content tampering, or timeliness doubts, the corresponding statements are decided to be removed or highlighted as warnings. When the global confidence score is lower than the preset second threshold or there is a core legal provision reference marked, the answer is rejected or transferred to human verification. Otherwise, a draft answer with a verification mark is output.

9. The method for resolving validity hierarchy and verifying legal citation tampering according to claim 1, characterized in that, The process also includes an offline phase before retrieval: the legal corpus is segmented into evidence fragments according to natural clauses or semantic paragraphs, and metadata records are created for each evidence fragment; The metadata record includes at least the unique identifier of the source document, the level of validity, the effective date, the repeal or revision status, the article number, and the start and end offsets of the characters. The level of validity, the effective date, and the repeal or revision status are automatically obtained by connecting to an authoritative legal database or by parsing legal texts using rules, and are managed in a versioned manner and maintained synchronously with the update of legal events.

10. A system for verifying the effectiveness hierarchy and statute of limitations and preventing tampering of legal citations, characterized in that, The method for resolving the validity hierarchy and statute of limitations and verifying the anti-tampering of legal citations as described in any one of claims 1 to 9 includes: The query rewriting module is used to classify intents and decompose questions based on users' legal Q&A queries, resulting in standardized search queries and intent tags; The hybrid retrieval module is used to sequentially classify intent, rewrite intent, and decompose questions to obtain standardized retrieval queries and their intent tags. Based on the standardized retrieval queries, candidate evidence fragments are retrieved from the legal corpus. Each candidate evidence fragment is associated with legal attribute information including at least the legal rank and the effective date. The reordering and validity conflict resolution module is used to sort the candidate evidence fragments by relevance and resolve conflicts of candidate evidence fragments based on validity rank and effective date, so as to determine the evidence set used to generate the answer. The constrained generation module is used to guide the large language model to perform constrained decoding based on the evidence set and generate a draft answer by using the standardized retrieval query, intent label and evidence set as constraints, and to annotate the evidence fragments cited in each statement in the draft answer. The dual verification module is used to perform dual verification on the draft answer, including semantic consistency verification and character-level word-by-word verification; The confidence calibration and decision output module is used to calculate the confidence level of the draft answer based on the results of the double verification, and decide whether to output the draft answer, trigger rejection, or transfer to human intervention based on the relationship between the confidence level and a preset threshold.

Images