Method for evaluating appropriateness of citations in legal documents and system using the same
The system addresses inefficiencies in verifying legal document citations by using AI to automatically evaluate their appropriateness, ensuring semantic, legal, and temporal consistency, thereby enhancing the reliability and reducing inaccuracies.
Patent Information
- Application Number
- KR1020260084616
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2026-05-11
- Publication Date
- 2026-11-11
- Estimated Expiration
- 2046-05-11
AI Technical Summary
Existing methods for verifying citations in legal documents are time-consuming and prone to inaccuracies, especially when using AI-based document generation, as they rely on manual searches and lack comprehensive evaluation of semantic, legal, and temporal consistency.
A method and system that automatically identifies and evaluates the appropriateness of citations by comparing the intent in legal documents with the original legal text, using an AI model that performs in-context learning to assess semantic consistency, legal consistency, factual consistency, temporal validity, and potential contradictions.
Enhances the efficiency and accuracy of verifying citations by automatically identifying and classifying their appropriateness, reducing the risk of hallucinations and improving the reliability of legal documents.
Smart Images

Figure 112026056725906-PAT00196_ABST
Abstract
Description
Technology Field
[0001] The present invention relates to a method for evaluating the appropriateness of citations within legal documents and a system utilizing the same. More specifically, the present invention relates to a legal document verification technology that identifies citations of precedents or statutes included in legal documents, searches for the original legal text corresponding to the identified citations, evaluates the appropriateness of the citations by comparing the intent of the citation in the legal document with the intent of the original legal text, and provides the evaluation results to a user. Background Technology
[0002] With the recent advancements in generative AI technology, AI-based document generation techniques are being utilized in the legal field for various document creation tasks, such as generating answers to legal inquiries, drafting legal reviews, reviewing contracts, drafting litigation documents, and writing legal articles. These AI-based techniques can improve work efficiency by enabling the rapid creation of documents through the rapid referencing of large volumes of legal data.
[0003] Unlike general natural language documents, legal documents often include citations of various legal sources, such as precedents, statutes, administrative rules, and authoritative interpretations. In legal documents, citations function not merely as reference information, but as grounds supporting specific legal conclusions or judgments. Therefore, if citations of precedents or statutes included in a legal document are inaccurate, used with a intent different from the original text, or based on statutes that are no longer in effect, the credibility of the entire document may be undermined.
[0004] Traditionally, to verify the accuracy of citations of precedents or statutes included in legal documents, users primarily employed a method involving separate searches for case numbers, precedent names, statute names, and article numbers, followed by direct viewing of the retrieved original texts to manually confirm whether the citations aligned with the document's conclusions or explanations. Since this method relies on the experience and judgment of legal professionals, reviewing such documents can be time-consuming, especially when the volume of the document or the number of citations is high. Furthermore, in cases where numerous similar cases or articles exist, there is a possibility that users may fail to accurately identify the appropriate original text or misunderstand the intent of the citation.
[0005] Meanwhile, while AI-based document generation technology can generate natural-sounding sentences in response to user queries, separate review is required to determine whether the cited precedents or statutes included in the generated sentences actually exist, or whether such citations substantially correspond to the argument of the document. Particularly in the legal field, not only the existence of precedents or statutes but also the rationale behind the rulings, the timing of the application of the statutes, whether the provisions have been amended, and the correspondence between legal issues and factual circumstances must be given significant consideration.
[0006] Accordingly, there is an increasing need for technology that supports users in more efficiently verifying precedents or statute citations included in legal documents and reviewing the reliability of such documents. The problem to be solved
[0007] The problem that the present invention aims to solve is to provide a method and system capable of automatically identifying citations of precedents or statutes included in legal documents, searching for the original legal text corresponding to the identified citations, and evaluating the appropriateness of the citations by comparing the intent for which the citations were used in the legal documents with the actual intent of the original legal text.
[0008] Furthermore, the problem that the present invention aims to solve is to provide a method and system capable of verifying whether a citation included in a legal document relates to an actual existing precedent or statute, and evaluating whether the cited statute corresponds to a repealed or amended statute, whether the cited precedent or statute was used in a manner contrary to the legal claims of the document, or whether there is a legal inconsistency between the context of the citation and the original text.
[0009] Furthermore, the problem that the present invention aims to solve is to provide a method and system for detecting inaccurate citations of precedents, fictitious citations of precedents, inappropriate citations of statutes, or citations that differ from the intent of the original text that may be included in legal text generated by generative artificial intelligence, and for providing the detection results so that a user can easily verify them.
[0010] Furthermore, the problem that the present invention aims to solve is to provide a method and system capable of evaluating citation appropriateness using an in-context learning method by utilizing an AI model whose parameters are not updated, and configuring an evaluation prompt that includes the context before and after a citation of a case law or statute, the original text of the corresponding law, criteria for judging citation appropriateness, and one or more examples.
[0011] Furthermore, the problem that the present invention aims to solve is to provide a method and system capable of evaluating the appropriateness of a citation by considering at least one of semantic consistency, legal consistency, factual consistency, temporal validity, and whether there is an opposing intent between a legal document and a legal original text, rather than merely comparing string or semantic similarity between a legal document and a legal original text.
[0012] Furthermore, the problem that the present invention aims to solve is to provide a method and system that can improve the review efficiency of legal document drafters or reviewers by classifying the results of a citation appropriateness evaluation into categories such as appropriate, caution, or requiring verification, and by visualizing and providing the citation section, corresponding original text section, and basis for judgment along with the classification results.
[0013] The problems of the present invention are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by a person skilled in the art from the description below. means of solving the problem
[0014] A method for evaluating the appropriateness of a citation within a legal document according to an embodiment of the present invention for solving the above problem comprises, in a method for evaluating the appropriateness of a citation within a legal document performed on a server, a legal text input step for receiving a legal text; a step for identifying a citation including a case precedent citation or a statute citation from the legal text and extracting the citation location and the context before and after the citation of the citation; a step for matching a legal text to search a legal original text corresponding to the citation from a legal database using at least one of a case precedent name, case number, court name, date of judgment, statute name, article number, and citation included in the citation; an analysis and evaluation step for deriving the intent of the citation to be supported by the citation from the context before and after the citation, deriving the intent of the original text from the legal original text, and comparing the intent of the citation with the intent of the original text to produce a result of evaluating the appropriateness of the citation for the citation; and a step for classifying and visualizing a verification result based on the result of evaluating the appropriateness of the citation to classify the citation into one of appropriate, caution, or need for verification, and providing the citation, the legal original text, and the result of evaluating the appropriateness of the citation to a user terminal by matching them.
[0015] Prior to the above legal text input step, the method may further include a legal text generation step that receives at least one of a user query, a document drafting request, case information, and basic materials, searches for relevant precedents or statutes from the above legal database, and generates the legal text by a search augmentation generation method.
[0016] In the above-mentioned case law and statute citation identification step, citation entities and contexts before and after the citation are extracted, and in the above-mentioned legal text matching step, when multiple original text candidates are found, a legal text is selected based on at least one of metadata consistency, contextual similarity, relevance to legal issues, and recency, and if a legal text is not found, the citation may be treated as a candidate for non-existence of original text or a citation requiring verification.
[0017] In the analysis and evaluation step above, an evaluation prompt including the citation, the context before and after the citation, the original legal text, criteria for judging citation appropriateness, and one or more examples is generated, and the evaluation prompt is input into an AI model in which parameters are not updated to produce the citation appropriateness evaluation result using an in-context learning method.
[0018] In the analysis and evaluation step described above, a set of legal propositions containing one or more legal propositions that the citation intends to support is generated from the context before and after the citation using an AI model and prompts for in-context learning, the legal original text is divided into multiple original text sections, and the semantic similarity, metadata consistency, and role score of each original text section between each legal proposition and each original text section are combined according to a reflection ratio weight to calculate a basis weight for each original text section, and the result of the citation appropriateness evaluation can be generated based thereon.
[0019] In the above analysis and evaluation step, for each legal proposition, the embedding vector of the original text section with the highest basis weight is calculated as the representative original text basis, and a multiple consistency vector can be calculated that includes semantic consistency, legal consistency, factual consistency, temporal validity score, authority score, and opposing intent risk score between the legal proposition and the representative original text basis.
[0020] In the above analysis and evaluation step, an appropriateness score for each legal proposition is calculated by applying coefficients to each element of the above multiple consistency vector and normalizing them with a sigmoid function, and an appropriateness score for each citation can be calculated by integrating multiple appropriateness scores for each proposition corresponding to the citation into a weighted combination of the average value and the minimum value.
[0021] In the analysis and evaluation step above, an evaluation prompt including the context before and after the citation, the representative source text basis, criteria for judging citation appropriateness, and one or more examples is input into an AI model in which parameters are not updated, and a label probability vector including the probability that the citation is classified as appropriate, caution, and need verification, respectively, is calculated using an in-context learning method, and the uncertainty of the label probability vector can be calculated through an entropy operation based on each probability value of the label probability vector.
[0022] In the analysis and evaluation step above, a final appropriateness score for the citation can be calculated by applying weights to and combining the appropriateness score for each citation, the probability of being classified as appropriate, the probability of being classified as cautionary, the probability of being classified as requiring verification, and the uncertainty, respectively.
[0023] A system for evaluating the appropriateness of citations within a legal document according to an embodiment of the present invention for solving the above problem comprises: a user terminal configured to receive a legal text or an evaluation request and display a result of evaluating the appropriateness of the citation; and a server configured to identify citations from the legal text, compare the intent of the citation with the intent of the original text to produce a result of evaluating the appropriateness of the citation, classify the citation as appropriate, caution, or need verification, and transmit the result to the user terminal. The server is configured to be connected to communicate with the user terminal and includes a database that stores original texts of case law, original texts of statutes, and metadata, and an AI model that performs citation identification, original text search, and citation appropriateness evaluation.
[0024] The above AI model may include a citation entity extraction model that extracts citation entities; a legal context analysis model that derives legal claims, requirements, effects, conclusions, and factual judgments from the context before and after the citation; a text intent analysis model that derives the intent of the ruling, requirements, legal effects, timing of application, whether it has been amended, and whether it has been abolished from the original legal text; and a citation appropriateness evaluation model that produces a citation appropriateness evaluation result by comparing the output of the legal context analysis model with the output of the text intent analysis model. Effects of the invention
[0025] According to an embodiment of the present invention, a citation of a precedent or statute included in a legal document is automatically identified, and after searching for the original legal text corresponding to the identified citation, the intent of the citation in the legal document can be compared with the actual intent of the original legal text, so that the user can more efficiently review the appropriateness of the citation included in the legal document.
[0026] In addition, since it is possible to comprehensively consider the existence of precedents or statutes, whether statutes have been repealed or amended, whether the intent of the citation matches the intent of the original text, and whether a contrary intent is cited, the reliability of legal documents can be verified more precisely compared to simple original text search or keyword matching methods.
[0027] In addition, since it is possible to detect inaccurate citations of precedents, fictitious citations of precedents, citations of precedents that differ from the intent of the original text, or citations of statutes that are no longer in effect that may be included in legal texts generated by generative AI, the risk of hallucinations that may occur during the AI-based legal document drafting process can be mitigated.
[0028] In addition, since the AI model can evaluate citation appropriateness using an in-context learning method without fine-tuning by utilizing evaluation prompts that include the citation context, corresponding legal text, judgment criteria, and examples, it can flexibly respond to various types of legal documents and citations without the need for separate large-scale model retraining.
[0029] Furthermore, since at least one of semantic consistency, legal consistency, factual consistency, temporal validity, authority, and risk of contradiction can be evaluated based on legal propositions extracted from the context of the citation and original grounds extracted from the original legal text, it is possible to determine whether the citation substantially conforms to the legal argument, going beyond merely whether a precedent or statute exists.
[0030] In addition, since the results of the citation appropriateness assessment can be classified as appropriate, caution, or requiring verification, and the cited section and the corresponding original text section can be visualized and provided together, legal professionals can quickly identify citations requiring review and reduce the time required for drafting and reviewing legal documents.
[0031] The effects according to the embodiments are not limited to those exemplified above, and a wider variety of effects are included in this specification. Brief explanation of the drawing
[0032] FIG. 1 is a schematic diagram illustrating the elements constituting a system for evaluating the appropriateness of citations within legal documents according to one embodiment of the present invention. Figure 2 is a flowchart schematically illustrating the entire service process provided through the system of Figure 1. Figure 3 is a flowchart illustrating the detailed steps constituting the evaluation step of Figure 2. Figure 4 is a diagram illustrating the functional elements of the AI model of Figure 1. Figure 5 is a schematic diagram illustrating the hardware configuration of the server of Figure 1. FIG. 6 is a diagram showing an application example in which the result of evaluating the appropriateness of a citation within a legal document according to one embodiment of the present invention is displayed on a user terminal. Specific details for implementing the invention
[0033] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims.
[0034] Although terms such as "first," "second," etc., are used to describe various components, it goes without saying that these components are not limited by these terms. These terms are used merely to distinguish one component from another. Therefore, it goes without saying that the first component mentioned below may be the second component within the technical scope of the present invention. A singular expression includes a plural expression unless the context clearly indicates otherwise.
[0035] In this specification, “legal text” may refer to all or part of a document containing legal judgments or legal arguments, such as a response to a legal inquiry, a legal review, a draft litigation document, a draft opinion, a contract review, or a legal news article. “Citation” may refer to an expression within the legal text that explicitly or implicitly refers to a precedent, statute, administrative rule, authoritative interpretation, or other legal material, and may include at least one of a precedent name, case number, court name, date of judgment, statute name, article number, article name, or citation.
[0036] In this specification, “legal text” may include at least one of the original text of case law, summary of case law, statutory provisions, amendment history of the statute, information on whether it has been repealed, or metadata thereof, which are retrieved from a database in response to a citation. “Purpose of citation” may refer to the legal argument, requirement, effect, conclusion, or judgment of factual circumstances that a specific citation in a legal text is intended to support, and “Purpose of original text” may refer to the intent of the ruling, intent of the provision, requirement of application, legal effect, or temporal validity derived from the original text of the law.
[0037] In this specification, “legal proposition” may refer to a unit legal argument or unit of judgment derived from the context before and after citation, and “original text section” may refer to at least one of a sentence, paragraph, article, clause, subparagraph, ruling, reasoning section, factual relationship section, or conclusion section constituting the original legal text. “Representative original text basis” may refer to an original text section among a plurality of original text sections that is calculated to have a high degree of consistency with the legal proposition, or the result of a weighted combination of original text sections.
[0038] In this specification, the “in-context learning method” may mean a method of producing an inference result by inputting an evaluation prompt, which includes a citation to be evaluated, context before and after the citation, a corresponding legal original text, a judgment criterion, and one or more examples, into an AI model without updating the parameters of the AI model.
[0039] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings. Identical or similar reference numerals are used for identical components in the drawings.
[0040] FIG. 1 is a schematic diagram illustrating the elements constituting a system for evaluating the appropriateness of citations within legal documents according to one embodiment of the present invention.
[0041] Referring to FIG. 1, a system for evaluating the appropriateness of citations within a legal document may include a server (10) and a user terminal (20). The server (10) is a hardware element for performing a method for evaluating the appropriateness of citations within a legal document and may be connected to the user terminal (20) via a wired or wireless communication network. The server (10) may receive a legal text from the user terminal (20), identify citations of precedents or statutes included in the received legal text, search for the original legal text corresponding to the identified citations, and generate a result of evaluating the appropriateness of citations by comparing the intent of the citation in the legal text with the intent of the original legal text.
[0042] The server (10) may include an AI model (11) and a database (12). The AI model (11) may be configured to extract citations of precedents or statutes included in legal text, analyze the context before and after the citation, analyze the intent of the original legal text, and evaluate the appropriateness of the citation. For example, the AI model (11) may include at least one of a citation entity extraction model, a legal context analysis model, an intent of the original text analysis model, and a citation appropriateness evaluation model. The AI model (11) may be implemented as a language model that operates in a context-in-context learning manner without fine-tuning, and in another embodiment, it may be implemented as a lightweight model optimized by a dataset for verifying legal citations.
[0043] The database (12) can store at least one of case law data, statute data, legal document data, metadata of case law and statute, amendment history of statutes, information on whether they have been repealed, case number of case law, date of judgment, name of court, and judgment details. Additionally, the database (12) can be used as a reference database for search augmented generation (RAG) to mitigate hallucinations during the process of generating legal text by generative AI. For example, the server (10) can search the database (12) for original texts of case law or statutes related to user queries or legal texts, generate legal text based on the searched reference materials, or verify the appropriateness of citations of the generated legal text retrospectively. Accordingly, the database (12) can support both primary hallucination reduction in the legal text generation stage and secondary hallucination reduction in the citation appropriateness evaluation stage.
[0044] The user terminal (20) may be a device for a user to input legal text or to check results generated or evaluated by the server (10). The user terminal (20) may be implemented as an electronic device capable of executing a desktop computer, a laptop computer, a tablet computer, a smartphone, or a web browser. The user terminal (20) may transmit legal text, legal inquiries, or evaluation requests to the server (10), and receive from the server (10) the results of the evaluation of the appropriateness of citing precedents or statutes, warnings, corresponding original texts, and the basis for evaluation, and display them on the screen.
[0045] The server (10), AI model (11), database (12), and user terminal (20) may be interconnected to sequentially perform identification of precedents or statute citations, search for original legal texts, analysis of the intent of the citations, evaluation of appropriateness, and provision of results for legal texts input or generated on the user terminal (20). With such a configuration, the user can efficiently verify not only whether a citation is included in a legal document, but also whether the citation corresponds to the intent of the actual original legal text.
[0046] Figure 2 is a flowchart schematically illustrating the entire service process provided through the system of Figure 1.
[0047] Referring to FIG. 2, a legal document citation appropriateness evaluation system according to one embodiment of the present invention may perform a legal text generation step (S100) and an evaluation step (S200). The legal text generation step (S100) may be a step in which a server (10) generates legal text based on a legal query, a document creation request, case information, or basic data input from a user terminal (20). In this case, the server (10) may generate a legal review document, a response document, an article, a draft opinion, a contract review document, or other legal documents using an AI model (11). Additionally, the server (10) may generate legal text using a search augmented generation (RAG) method by referencing precedents, statutes, legal documents, or metadata stored in a database (12). Accordingly, the legal text generation step (S100) may support primary hallucination reduction during the generation process.
[0048] However, in the present invention, the legal text generation step (S100) can be understood as a preliminary step for evaluating the appropriateness of citations within a legal document, and the technical features of the present invention are not limited to the generation of the legal document itself. That is, the present invention can be applied not only to legal text directly generated by the server (10), but also to externally generated legal text input from the user terminal (20), legal documents written by the user, or legal text generated by another generative artificial intelligence system.
[0049] The evaluation step (S200) is a step for retrospectively verifying the citation of precedents or statutes included in the legal text generated in the legal text generation step (S100) or input from the user terminal (20). Specifically, the server (10) identifies the citation of precedents or statutes included in the generated legal text, searches for the original legal text corresponding to the identified citation in the database (12), and can compare the intent for which the citation was used in the legal text with the actual intent of the original legal text. Subsequently, the server (10) can generate a result of evaluating the appropriateness of the citation by considering at least one of the following: whether the citation exists, whether the intent matches the original text, whether the opposite intent is used, whether it is a repealed or amended statute, and whether verification is required.
[0050] Accordingly, the entire service process of Fig. 2 may have a two-stage structure in which, after generating a legal document, the citations of precedents and statutes within the generated legal text are verified again. According to this structure, the possibility of hallucination can be primarily reduced through search augmented generation (RAG) in the legal text generation stage (S100), and a secondary hallucination suppression function can be provided by retrospectively verifying the appropriateness of the citations in the generated legal text in the evaluation stage (S200). Accordingly, the user can verify not only the sentences of the generated legal document themselves, but also whether the citations of precedents or statutes included in the sentences are supported by the actual original legal text.
[0051] Figure 3 is a flowchart illustrating the detailed steps constituting the evaluation step of Figure 2.
[0052] Referring to FIG. 3, the evaluation step (S200) may include a legal text input step (S210), a case law / statute citation identification step (S220), a legal original text matching step (S230), an analysis and evaluation step (S240), and a verification result classification and visualization step (S250).
[0053] The legal text input step (S210) may be a step in which the server (10) receives legal text from a user terminal (20) or receives legal text generated in the legal text generation step (S100) as an evaluation target. The legal text may be a document containing a response to a legal inquiry, a legal review document, a draft of a litigation document, a draft of an opinion, a contract review document, a legal news article, or other legal judgments. In one embodiment, the legal text may be text generated by the AI model (11) of the server (10), and in another embodiment, it may be text created by an external generative AI system or a user and input through the user terminal (20). Accordingly, the present invention is not limited to the generation of legal documents itself, but may be implemented as a structure that retrospectively verifies precedents or statute citations included in already generated legal text.
[0054] The step of identifying citations of precedents and statutes (S220) may be a step in which the server (10) identifies citations regarding precedents or statutes from input legal text. Specifically, the server (10) may use an AI model (11) or a separate citation identification algorithm to extract at least one of the following: a precedent name, case number, date of judgment, court name, statute name, article number, article name, citation, location of citation, and context before and after the citation included in the legal text. For example, the server (10) may identify expressions such as "Supreme Court Judgment 20XX.XX.XX. 20XXdaXXXXX", "Article XXX of the Civil Act", and "Article XX of the Patent Act" as citations of precedents or statutes. Additionally, even in the absence of an explicit case number or article number, the server (10) may identify expressions referring to a specific precedent or statute from the context of the legal text as candidate citations.
[0055] The legal text matching step (S230) may be a step of searching for and matching legal texts corresponding to the citations identified in the case law / statute citation identification step (S220) in the database (12). The database (12) may include the original text of a case law, a summary of a case law, case law metadata, the original text of a statute, article information, the effective date of the statute, amendment history, and information on whether it has been repealed. The server (10) may identify the corresponding legal text by comparing the case number, case law name, court name, date of judgment, statute name, article number, or keyword of the identified citation with the metadata stored in the database (12). At this time, if the corresponding legal text is not found, the server (10) may process the citation as a candidate for non-existence of original text or a citation requiring verification. Additionally, if multiple corresponding original text candidates are found, the server (10) may select one or more original text candidates based on at least one of metadata consistency, contextual similarity, relevance to legal issues, or recency.
[0056] The analysis and evaluation step (S240) may be a step of evaluating the appropriateness of a citation by comparing the intent for which an identified precedent or statute citation was used in the legal text with the actual intent of the original legal text. Specifically, the server (10) may analyze the legal argument, requirement, effect, conclusion, or factual determination that the citation is intended to support from the context before and after the citation. Additionally, the server (10) may analyze the intent of the precedent's ruling, the requirements and effects of the statute's provisions, the timing of application, and whether the citation has been amended or repealed from the original legal text. By comparing the intent of the citation analyzed in this way with the intent of the original text, the server (10) may determine whether the citation aligns with the argument of the legal text, whether it was used with a different intent from the original text, whether it was used with an opposing intent, or whether it was used based on a statute that is not currently valid. In one embodiment, the analysis and evaluation step (S240) may be performed by an AI model (11) using an in-context learning method without fine-tuning. In this case, the server (10) can generate an evaluation prompt including a citation section, context before and after the citation, the original text of the corresponding law, judgment criteria, and one or more examples, and input the evaluation prompt into an AI model (11) to produce a result of evaluating the appropriateness of the citation.
[0057] The verification result classification and visualization step (S250) may be a step of classifying and visualizing the results of the analysis and evaluation step (S240) into a form that can be provided to the user. The server (10) may generate at least one verification result for each citation among appropriate, caution, need for verification, original text not existing, repealed statute, citation with a counter-intent, or inconsistency in legal principles. Additionally, the server (10) may generate visualization data that corresponds the citation section within the legal text, the corresponding original text section of the law, the basis for judgment, the reason for caution, and the verification label to each other so that they can be displayed on the user terminal (20). Accordingly, the user terminal (20) may highlight the citation requiring review, display a comparison between the quoted sentence and the corresponding original text, or provide the reason for caution in the form of a notification.
[0058] In this way, the evaluation step (S200) of Fig. 3 is a procedure for retrospectively verifying precedents or statute citations included in the generated legal text, and it can improve the reliability of the legal text by evaluating the correspondence between the context of the citation and the intent of the original legal text, rather than merely searching for and providing the original legal text.
[0059] In one embodiment, the analysis and evaluation step (S240) may include a multi-axis legal citation evaluation step that extracts a set of legal propositions from the citation context and calculates the consistency between the extracted set of legal propositions and multiple original text sections within the original legal text. This embodiment is intended not merely to determine whether a citation within the legal text refers to an actual existing precedent or statute, but to evaluate whether the specific legal proposition that the citation is intended to support within the legal text is substantially supported by the original legal text.
[0060] Specifically, the server (10) inputs the legal text The i-th citation identified in the context before and after can be extracted. Here, is a legal text subject to evaluation, and is the i-th citation regarding precedents or statutes, and is a quote It may be contextual information including a sentence containing, preceding and succeeding sentences, paragraphs, a conclusion sentence, or a relationship with a user query.
[0061] The server (10) is an AI model Using the context before and after Quoted from One or more legal propositions that it intends to support A set of legal propositions including It can be calculated (Mathematical Formula 1).
[0062] <Mathematical Formula 1>
[0063]
[0064] In mathematical formula 1, is the i-th quotation It is a set of legal propositions corresponding to, and is the j-th legal proposition, and r is a citation It is the number of legal propositions extracted in correspondence with. is a parameter It is an AI model that has, is a quote , context , may be a prompt for in-context learning including criteria for extracting legal propositions and one or more examples. In this case, the AI model is a set of legal propositions based on the context and examples included in the prompt, without parameter updates by fine-tuning. It can produce.
[0065] In addition, the server (10) retrieves the legal text from the legal text matching step (S230). plural original text sections It can be divided into. Here, is the i-th quotation It is the original text of a precedent or statute found in correspondence with, and is the original legal text It is the k-th original text segment constituting, and m is the number of original text segments. Original text segment It may correspond to at least one of the judgment, reasoning, legal principles, factual circumstances, conclusion, the main text, paragraph, subparagraph, or supplementary provisions of a statute.
[0066] Server (10) is a legal proposition and multiple original text sections Evidence weights for each text segment based on semantic similarity, metadata consistency, and role scores of the text segments It can be calculated (Mathematical Formula 2).
[0067] <Mathematical Formula 2>
[0068]
[0069] In mathematical formula 2, is the number of original text sections generated by dividing the above legal original text, and is 1 or more The k-th original text section among the plurality of original text sections that is subject to the calculation of the basis weight, as an integer below. It is an index that identifies, and It starts from 1 It is a summing index for summing the above multiple original text sections up to, and is a legal proposition Regarding the k-th original text section It is a basis weight indicating the extent to which it can be used as a basis. is a legal proposition and original text section It is the semantic similarity between, is a metadata consistency based on at least one of a case number, precedent name, court name, date of judgment, statute name, or article number, and is the original text section This is a role score that reflects which part corresponds to the explanation of legal principles, conclusion, judgment, factual circumstances, divisive arguments, or additional explanation. and is a weight that adjusts the reflection ratio of semantic similarity, metadata consistency, and role scores. Mathematical Equation 2 is a legal proposition among multiple source text segments The purpose is to give greater weight to sections that are highly likely to practically support it.
[0070] Server (10) provides the basis weights for each original text section Legal propositions based on Representative original source basis corresponding to It can be calculated (Mathematical Formula 3).
[0071] <Mathematical Formula 3>
[0072]
[0073] In mathematical formula 3, is a legal proposition It is the representative original source corresponding to, and is the original text section This is the output of an embedding function that converts into a semantic vector. Accordingly, the server (10) can evaluate the appropriateness of citation based on sections of the original text that are highly relevant to the relevant legal proposition, rather than comparing the entire original text of the law collectively.
[0074] In one embodiment, the analysis and evaluation step (S240) is a legal proposition and representative original text basis A multiple consistency vector including at least one of semantic consistency, legal consistency, factual consistency, temporal validity score, authority score, and dissenting opinion risk score. It can be calculated (Mathematical Formula 4).
[0075] <Mathematical Formula 4>
[0076]
[0077] In mathematical formula 4, legal proposition and representative original text basis It is the degree of semantic consistency between, It is a degree of legal consistency that indicates the extent to which a legal proposition is supported by the legal principles or intent of the provision in the original text. is a degree of factual consistency indicating the extent to which the factual relationships in the quotation context correspond to the factual relationships in the source text, and It is a temporal validity score indicating whether a statute or case law is valid at the reference point. is an authority score reflecting court levels, statute hierarchy, recency, or relevance, and is a conflict risk score indicating the degree to which the intent of the citation is contrary to the intent of the original text.
[0078] The server (10) is a multiple alignment vector Legal propositions based on Propositional appropriateness scores for It can be calculated (Mathematical Formula 5).
[0079] <Mathematical Formula 5>
[0080]
[0081] In mathematical formula 5, is the j-th legal proposition It is an appropriateness score for, is a sigmoid function that normalizes input values to values between 0 and 1. is the bias value, and inside is a coefficient that adjusts the reflection ratio of each evaluation element. is a quote This is the risk value that there is no corresponding original legal text. Mathematical formula 5 is intended to lower the appropriateness score even for semantically similar citations if they are not legally consistent, differ in factual circumstances, are not currently valid, or are used with the opposite intent.
[0082] One quote a multiple legal proposition In the case of supporting, the server (10) has multiple propositional appropriateness scores Quote by combining Appropriateness score by citation for It can be calculated (Mathematical Formula 6).
[0083] <Mathematical Formula 6>
[0084]
[0085] In mathematical formula 6, is a quote It is the appropriateness score per citation for, and is a coefficient that adjusts the reflection ratio of the lowest score among multiple propositional appropriateness scores. According to mathematical formula 6, if a single citation supports multiple legal propositions, the appropriateness score of the entire citation may be lowered if some legal propositions are not supported by the original legal text. Accordingly, the server (10) can evaluate complex citations that are appropriate in some contexts but inappropriate in others more conservatively.
[0086] In one embodiment, the analysis and evaluation step (S240) is an evaluation prompt configured in an in-context learning method. AI model Input to determine label probabilities for appropriate, caution, and verification needs. It can be calculated (Mathematical Formula 7).
[0087] <Mathematical Formula 7>
[0088]
[0089] In mathematical formula 7, is a quote , quotation context , set of legal propositions , Representative source text basis It is an evaluation prompt that includes judgment criteria and one or more examples. is the appropriate probability, and is the probability of attention, and is the probability that needs to be verified. In this case, the AI model parameters of is not updated during the evaluation process, and the AI model is the evaluation prompt Label probability based on the context, source grounds, and examples included in It can produce.
[0090] Server (10) has a citation appropriateness score , label probability and uncertainty of label probability Final appropriateness score based on It can be calculated (Equation 8, Equation 9).
[0091] <Mathematical Formula 8>
[0092]
[0093] <Mathematical Formula 9>
[0094]
[0095] In mathematical formula 8, is the label probability It is an uncertainty defined by the entropy of, and is a set of labels including appropriate, caution, and need for verification, and is the probability for label y. In Equation 9, is a quote It is the final adequacy score for, inside is a weight that adjusts the reflection ratio of the appropriateness score, appropriateness probability, caution probability, verification need probability, and uncertainty per citation. According to Equations 8 and 9, if the AI model (11) fails to clearly select a specific label, the final appropriateness score Since the value may be lowered, the server (10) may classify the uncertain citation as needing verification or as a subject of caution.
[0096] The verification result classification and visualization step (S250) is the final appropriateness score , original non-existent risk value Citation based on at least one of whether the statute is repealed or amended, and the risk score against intent. It can be classified as appropriate, requiring caution, or requiring verification. For example, the server (10) quotes If the corresponding original text of the law does not exist, if the cited statute has been repealed, or if the risk score against intent is above the threshold value, the final adequacy score Regardless of this, the citation can be classified as a subject of caution. With this configuration, the server (10) can evaluate the appropriateness of the citation by comprehensively reflecting the existence of precedents or statute citations included in the generated legal text, legal consistency, correspondence with facts, temporal validity, and uncertainty of judgment by the AI model.
[0097] Figure 4 is a diagram illustrating the functional elements of the AI model of Figure 1.
[0098] Referring to FIG. 4, the AI model (11) may include a citation entity extraction model (310), a legal context analysis model (320), a source text intent analysis model (330), and a citation appropriateness evaluation model (340). The AI model (11) may be included in and executed on the server (10) of FIG. 1, and may be implemented by the processor of the server (10) executing program instructions stored in memory. Additionally, each detailed model of the AI model (11) may be implemented as a plurality of physically separated models, or may be functionally separated and implemented by different prompts or different task modules within a single large-scale language model.
[0099] The citation entity extraction model (310) is a model for identifying precedents or statute citations from input legal text. The citation entity extraction model (310) can extract at least one of the following: precedent name, case number, court name, date of judgment, statute name, article number, article name, citation, and citation location included in the legal text. For example, the citation entity extraction model (310) can extract expressions such as “Supreme Court Judgment ○○○○.○○.○○. ○○da○○○○”, “Article ○○○○ of the Civil Act”, and “Article ○○ of the Patent Act” from the legal text as precedent or statute citation entities. In one embodiment, the citation entity extraction model (310) may be implemented as a Named Entity Recognition model, a Sequence Labeling model, a Transformer Encoder-based model, or a hybrid model combining a regular expression-based candidate extraction module and an AI classification model.
[0100] The legal context analysis model (320) is a model for analyzing the purpose for which a citation of a precedent or statute extracted by the citation entity extraction model (310) is used within a legal text. The legal context analysis model (320) can derive a legal claim, requirement, effect, conclusion, or factual determination that the citation is intended to support, based on at least one of the sentence containing the citation, preceding and succeeding sentences, paragraph, user query, document type, or conclusion sentence. For example, the legal context analysis model (320) can analyze whether a specific precedent citation was used to support conditions for contract termination, whether liability for damages is established, interpretation of the scope of rights, or procedural requirements. In one embodiment, the legal context analysis model (320) may be implemented as an in-context inference model using a text classification model, a natural language inference model, a document embedding model, or a large language model.
[0101] The original text intent analysis model (330) is a model for analyzing the actual intent of a case law or a statute text retrieved from a database (12). The original text intent analysis model (330) can analyze at least one of the judgment, reasoning, ruling details, factual circumstances, legal principles, conclusion, or the main text, paragraph, subparagraph, supplementary provisions, effective date, and amendment history of a statute. The original text intent analysis model (330) can select a section of the original text that is highly relevant to the citation context among the original texts of the statutes, and can derive actual legal principles, application requirements, legal effects, grounds for exception, or temporal validity from the selected section of the original text. In one embodiment, the original text intent analysis model (330) can be implemented as a Retrieval-Augmented Generation model, a Long-Context model, a Summarization model, a Semantic Search model, or an Embedding-based Ranking model.
[0102] The citation appropriateness evaluation model (340) is a model for evaluating the appropriateness of citing a precedent or statute by comparing the intent of the citation derived by the legal context analysis model (320) with the intent of the original text derived by the original text intent analysis model (330). The citation appropriateness evaluation model (340) can produce an evaluation result by considering at least one of whether the intent of the citation and the intent of the original text are logically or legally consistent, whether the citation is used with an intent contrary to the original text, whether the cited statute is a repealed or amended statute, whether a corresponding original text does not exist, or whether the factual circumstances substantially correspond. The evaluation result may be output as at least one label among appropriate, caution, need for verification, original text not existing, repealed statute, legal inconsistency, or citation with an intent contrary. In one embodiment, the citation appropriateness evaluation model (340) may be implemented as a natural language inference model, a cross-encoder-based sentence pair determination model, a few-shot evaluation model using a large-scale language model, or a lightweight model optimized by a legal citation verification dataset.
[0103] The citation entity extraction model (310), legal context analysis model (320), original text intent analysis model (330), and citation appropriateness evaluation model (340) can be linked sequentially. Specifically, the citation entity extraction model (310) extracts citations to be verified from legal texts, the legal context analysis model (320) analyzes the contextual intent of the extracted citations, the original text intent analysis model (330) analyzes the actual intent of the legal original text matched from the database (12), and the citation appropriateness evaluation model (340) evaluates the correspondence between the intent of the citation and the intent of the original text. Accordingly, the AI model (11) can evaluate whether the citation substantially corresponds to the legal arguments of the document, going beyond the simple existence of a citation of a precedent or statute within the legal text.
[0104] In one embodiment, the detailed models (310, 320, 330, 340) of the AI model (11) may be implemented by calling the same AI model with different prompts. In this case, the server (10) may generate a prompt for citation extraction, a prompt for context analysis, a prompt for original text intent analysis, and a prompt for citation appropriateness evaluation, respectively, and perform an evaluation procedure using an in-context learning method by sequentially inputting them into the AI model, the parameters of which are not updated. In another embodiment, some of the detailed models (310, 320, 330, 340) may be implemented as a rule-based algorithm or a search algorithm, and the remaining parts may be implemented as an AI model.
[0105] Figure 5 is a schematic diagram illustrating the hardware configuration of the server of Figure 1.
[0106] Referring to FIG. 5, the server (10) may include a processor (110), memory (120), a transmission / reception device (130), an input interface device (140), an output interface device (150), a storage device (160), and a bus (170). The server (10) may be a computing device for performing full service provision, legal text generation, identification of precedent / statute citations, legal original text matching, analysis and evaluation, and classification and visualization processing of verification results in the legal document citation appropriateness evaluation system of FIG. 1.
[0107] A processor (110) can perform each step of the method for evaluating the appropriateness of citations within a legal document by controlling the operation of the server (10) and executing program instructions stored in memory (120) or storage device (160). For example, the processor (110) can identify citations of precedents or statutes from legal text, search for original texts of precedents or statutes stored in the database (12), execute an AI model (11) to analyze the context of the citation and the intent of the original text, and produce a result of evaluating the appropriateness of the citation.
[0108] The memory (120) can temporarily store program instructions executed by the processor (110), intermediate data generated during operation, evaluation prompts, citation context information, and legal original text information. The memory (120) may include ROM and RAM, the ROM stores programs for the basic operation of the server (10), and the RAM can temporarily store data required during the legal text processing and the inference process of the AI model (11).
[0109] The transmitting and receiving device (130) may be a communication module for transmitting and receiving data to and from a user terminal (20), an external legal data providing server, or another system. The transmitting and receiving device (130) may receive legal text, a query, or an evaluation request from the user terminal (20) and transmit the citation appropriateness evaluation result, caution information, corresponding original text, and visualization data to the user terminal (20).
[0110] The input interface device (140) may be a device for receiving operation settings, evaluation criteria, model setting values, database update information, or user input of the server (10) from an administrator or an external device. The output interface device (150) may be a device for outputting the processing status, evaluation results, error information, or reports for the administrator of the server (10) to the outside.
[0111] The storage device (160) can store a program for performing a method to evaluate the appropriateness of citations within legal documents, a model file of an AI model (11), a database of precedents and statutes, metadata, evaluation history, user feedback, and log data. In one embodiment, the storage device (160) may constitute all or part of the database (12) of FIG. 1, and in another embodiment, the database (12) may be implemented as a separate storage server capable of communicating with the server (10).
[0112] The bus (170) may be an internal communication path that interconnects the processor (110), memory (120), transmission / reception device (130), input interface device (140), output interface device (150), and storage device (160) to transmit data and control signals. Accordingly, the server (10) can perform legal text input, identification of precedent / legal citations, original text matching, AI-based analysis and evaluation, and provision of verification results as a series of processing flows through the interoperability of each hardware component.
[0113] FIG. 6 is a diagram showing an application example in which the result of evaluating the appropriateness of a citation within a legal document according to one embodiment of the present invention is displayed on a user terminal.
[0114] Referring to FIG. 6, the user terminal (20) may display a citation appropriateness evaluation result area (540) along with legal text. The citation appropriateness evaluation result area (540) may be a screen area for providing the user with the verification results of a precedent or statute citation generated by the server (10) using the AI model (11) of FIG. 4. Specifically, the server (10) may extract the precedent name, case number, statute name, or article number included in the legal text using a citation entity extraction model (310), and analyze the purpose for which each citation was used in the legal text using a legal context analysis model (320). Additionally, the server (10) may analyze the purpose of the original text of the precedent or statute retrieved from the database (12) using an original text purpose analysis model (330), and evaluate whether the citation context corresponds to the purpose of the original text using a citation appropriateness evaluation model (340).
[0115] The area of the citation appropriateness evaluation result (540) may include multiple citation items. Each citation item may include a citation of a precedent or statute identified in the legal text, an evaluation label for the citation, and an explanation of the basis for the evaluation. For example, the first evaluation label (541) may be an “appropriate” label displayed when the citation is consistent with the intent of the citation in the legal text and the intent of the original text of the law. A citation item with the first evaluation label (541) displayed may be provided when the relevant precedent or statute is evaluated as a citation that supports the conclusion or explanation of the legal text.
[0116] On the other hand, the second evaluation label (542) may be a “Caution” label displayed when there is a difference between the intent of the citation in the legal text and the intent of the original legal text, or when there is low relevance to the intent of the citation, or when additional review is required regarding legal or factual circumstances. A citation item with the second evaluation label (542) displayed may indicate that the user needs to review it further by comparing the original text and the context of the citation. In one embodiment, the “Caution” label may be displayed when there is no corresponding original text, when the cited statute is a repealed or amended statute, when a precedent or statute is used with the opposite intent, or when the degree of consistency between the context of the citation and the intent of the original text is below a threshold value.
[0117] In one embodiment, the citation appropriateness evaluation result area (540) may display identification information of a precedent or statute, an evaluation label, the reason for evaluation, the result of comparison with the original text, and guidance text for user review for each citation item. Accordingly, the user can intuitively distinguish between reliable citations and citations requiring further review among the precedent or statute citations included in the generated legal text. Additionally, the user terminal (20) may display an expanded version of the original text of the precedent, the statute provision, the relevant paragraph, or the basis for comparison corresponding to the citation in response to user input selecting an evaluation label or citation item.
[0118] According to this configuration, the server (10) does not merely store the citation appropriateness evaluation results produced by the AI model (11) as simple scores or internal judgment values, but can also provide them as visualized verification results through the user terminal (20). Therefore, while reviewing the entire content of the generated legal text, the user can check the appropriateness of the citation of precedents or statutes item by item and efficiently select citations that are inappropriate or require verification.
[0119] Although embodiments according to the technical concept of the present invention have been described above with reference to the attached drawings, those skilled in the art will understand that the present invention may be implemented in other specific forms without changing its technical concept or essential features. The embodiments described above should be understood as illustrative in all respects and not restrictive. Explanation of the symbols
[0120] 10: Server 11: AI Model 12: Database 20: User Terminal 310: Citation Entity Extraction Model 320: Legal Context Analysis Model 330: Original Text Purpose Analysis Model 340: Citation Appropriateness Assessment Model 540: Citation Appropriateness Assessment Results Area 541: First evaluation label 542: Second evaluation label
Claims
Claim 1 A method for evaluating the appropriateness of citations within legal documents performed on a server comprises: a legal text input step for receiving legal text; a case law / statute citation identification step for identifying citations including case law citations or statute citations from the legal text and extracting the citation location and context before and after the citation; a legal text matching step for searching a legal original text corresponding to the citation from a legal database using at least one of a case law name, case number, court name, date of judgment, statute name, article number, and citation included in the citation; and an analysis and evaluation step for deriving the intent of the citation to be supported by the citation from the context before and after the citation, deriving the intent of the original text from the legal original text, and comparing the intent of the citation with the intent of the original text to produce a result of evaluating the appropriateness of the citation for the citation. and includes a verification result classification and visualization step of classifying the citation into one of appropriate, caution, or needing verification based on the citation appropriateness evaluation result, and providing the citation, the legal original text, and the citation appropriateness evaluation result to a user terminal by correlating them with each other, wherein the analysis and evaluation step comprises calculating a set of legal propositions including one or more legal propositions that the citation intends to support from the context before and after the citation, dividing the legal original text into multiple original text sections, and calculating the citation appropriateness evaluation result based on the degree of consistency between the one or more legal propositions and the multiple original text sections, wherein the set of legal propositions is calculated based on the following mathematical formula 1, and <Mathematical Formula 1> Based on the following mathematical formula 2, calculate the basis weight for each original text section between the above legal proposition and the above plurality of original text sections, and <Mathematical Formula 2> In the above mathematical formula 1, is the i-th quotation It is a set of legal propositions corresponding to, and is the j-th legal proposition, and r is a citation It is the number of legal propositions extracted in correspondence with, and is a parameter It is an AI model that has, is a quote , context , a prompt for in-context learning including criteria for extracting legal propositions and one or more examples, and in the above mathematical formula 2, is the number of original text sections generated by dividing the above legal original text, and is 1 or more The k-th original text section among the plurality of original text sections that is subject to the calculation of the basis weight, as an integer below. It is an index that identifies, and It starts from 1 It is a summing index for summing the above multiple original text sections up to, and is a legal proposition Regarding the k-th original text section It is a basis weight indicating the extent to which it can be used as a basis, and is a legal proposition and original text section It is the semantic similarity between, is a metadata consistency based on at least one of a case number, precedent name, court name, date of judgment, statute name, or article number, and is the original text section It is a role score that reflects which part corresponds to the explanation of legal principles, conclusion, judgment, factual circumstances, circumstantial evidence, or additional explanation, and and A method for evaluating the appropriateness of citations within legal documents, which is a weighting factor that adjusts the reflection ratios of semantic similarity, metadata consistency, and role scores. Claim 2 A method for evaluating the appropriateness of citations within a legal document according to claim 1, further comprising, prior to the legal text input step, a legal text generation step that receives at least one of a user query, a document drafting request, case information, and basic materials, searches for precedents or statutes related to the user query, the document drafting request, the case information, or the basic materials from the legal database, and generates the legal text by a search-augmented generation method that uses the searched precedents or statutes as reference materials, wherein the legal text input step receives the legal text generated in the legal text generation step as an evaluation target. Claim 3 A method for evaluating the appropriateness of citations within a legal document, wherein the step of identifying the citation of a precedent or statute extracts at least one of a precedent name, case number, court name, date of judgment, statute name, article number, article name, and citation from the legal text as a citation entity, and extracts at least one of a sentence containing the citation entity, sentences before and after the citation entity, a paragraph containing the citation entity, and a conclusion sentence of the legal text as the context before and after the citation, and the step of matching the legal text selects the legal text based on at least one of metadata consistency, contextual similarity, legal issue relevance, and recency between the citation and each original text candidate when multiple original text candidates corresponding to the citation are found, and processes the citation as a candidate for non-existence of original text or a citation requiring verification. Claim 4 A method for evaluating the appropriateness of a citation within a legal document, wherein the analysis and evaluation step comprises generating an evaluation prompt including the citation, the context before and after the citation, the original legal text, criteria for judging the appropriateness of the citation, and one or more examples, and inputting the evaluation prompt into an AI model whose parameters are not updated to produce the evaluation result of the appropriateness of the citation using an in-context learning method. Claim 5 delete Claim 6 In claim 1, the analysis and evaluation step calculates a representative original text basis corresponding to the legal proposition based on the following mathematical formula 3, and <Mathematical Formula 3> Based on the following mathematical formula 4, a multiple consistency vector between the above legal proposition and the above representative original text basis is calculated, and <Mathematical Formula 4> In the above mathematical formula 3, is the above legal proposition It is the representative original source corresponding to, and is the above original text section It is the output of an embedding function that converts into a meaning vector, and in the above mathematical formula 4, is the above legal proposition and the basis of the above representative original text The multiple matching vector between them is, and The above legal proposition and the basis of the above representative original text It is the degree of semantic consistency between, ...is a degree of legal consistency indicating the extent to which a legal proposition is supported by the legal principles or intent of the provision in the original text, and is a degree of factual consistency indicating the extent to which the factual relationships in the quotation context correspond to the factual relationships in the source text, and is a temporal validity score indicating whether a statute or case law principle is valid at the reference point, and is an authority score reflecting court levels, statute hierarchy, recency, or relevance, and A method for evaluating the appropriateness of citations within legal documents, which is a contra-intent risk score indicating the degree to which the intent of the citation is contrary to the intent of the original text. Claim 7 In Clause 6, the analysis and evaluation step calculates a proposition-specific appropriateness score for the legal proposition based on the following mathematical formula 5, and <Mathematical Formula 5> Calculate the appropriateness score for each citation for the above citations based on the following mathematical formula 6, and <Mathematical Formula 6> In the above mathematical formula 5, is the j-th legal proposition It is an appropriateness score for, is a sigmoid function that normalizes input values to values between 0 and 1, and is the bias value, and inside is a coefficient that adjusts the reflection ratio of each evaluation element, and The above citation It is the risk value that the original legal text corresponding to does not exist, and in the above mathematical formula 6, is a quote It is the appropriateness score per citation for, and is the propositional appropriateness score for the j-th legal proposition corresponding to the i-th citation, and r is the number of legal propositions extracted corresponding to the i-th citation, and A method for evaluating the appropriateness of citations within legal documents, which is a coefficient that adjusts the reflection ratio of the lowest score among multiple propositional appropriateness scores. Claim 8 In claim 7, the analysis and evaluation step calculates a label probability vector for the citation from an AI model based on the following Equation 7, and <Equation 7> Calculate the uncertainty of the above label probability vector based on the following Equation 8, and <Equation 8> In the above mathematical formula 7, is the label probability vector for the i-th citation above, and is the probability that the above i-th citation is classified as appropriate, and is the probability that the above i-th citation is classified as a caution, and is the probability that the above i-th citation is classified as needing verification, and is a parameter It is an AI model that has, is an evaluation prompt comprising the above i-th citation, the context before and after the citation of the above i-th citation, a set of legal propositions corresponding to the above i-th citation, a representative source text basis corresponding to the above legal propositions, criteria for judging citation appropriateness, and one or more examples, and in the above mathematical formula 8, is the uncertainty of the label probability vector for the i-th citation above, and is a set of labels including appropriate, caution, and need for verification, and is the probability for label y included in the above label set, and the AI model receives the above evaluation prompt as input and calculates the above label probability vector, wherein the parameters of the AI model in the above analysis and evaluation step A method for evaluating the appropriateness of a citation within a legal document, wherein the AI model is not updated, and the AI model calculates the label probability vector using an in-context learning method based on the context before and after the citation included in the evaluation prompt, the representative source text basis, the criteria for judging the appropriateness of the citation, and one or more examples. Claim 9 In claim 8, the analysis and evaluation step calculates a final appropriateness score for the citation based on the following mathematical formula 9, and <Mathematical Formula 9> In the above mathematical formula 9, is the final appropriateness score for the above i-th citation, and is the citation appropriateness score for the above i-th citation, and inside A method for evaluating the appropriateness of citations within legal documents, wherein the weights are the appropriateness score for each citation, the probability of being classified as appropriate, the probability of being classified as cautionary, the probability of being classified as requiring verification, and the reflection ratio of uncertainty. Claim 10 A system for evaluating the appropriateness of citations within legal documents comprises: a user terminal configured to receive a legal text or an evaluation request and to display the results of an evaluation of the appropriateness of citations for case law citations or statute citations included in the legal text; and a server configured to be communicably connected to the user terminal and to perform an evaluation of the appropriateness of citations for the legal text, wherein the server comprises a database storing at least one of the original text of a case law, the original text of a statute, case law metadata, statute metadata, and information on the amendment history and repeal status of the statute. and includes an AI model configured to be executed by a processor of the server to identify a citation of a precedent or a statute included in the legal text, search for a legal original text corresponding to the precedent citation or the statute citation, and evaluate the appropriateness of the citation for the precedent citation or the statute citation, wherein the server receives the legal text or the evaluation request from the user terminal, identifies a citation including a precedent citation or a statute citation from the legal text using the AI model, extracts the context before and after the citation of the citation, searches for a legal original text corresponding to the citation in the database, compares the intent of the citation derived from the context before and after the citation with the intent of the original text derived from the legal original text to produce a result of evaluating the appropriateness of the citation for the citation, produces a set of legal propositions including one or more legal propositions that the citation is intended to support from the context before and after the citation, divides the legal original text into multiple original text sections, and produces the result of evaluating the appropriateness of the citation based on the degree of consistency between the one or more legal propositions and the multiple original text sections, wherein the set of legal propositions is produced based on the following mathematical formula 1, and <mathematical formula 1> Based on the following mathematical formula 2, calculate the basis weight for each original text section between the above legal proposition and the above plurality of original text sections, and <Mathematical Formula 2> In the above mathematical formula 1, is the i-th quotation It is a set of legal propositions corresponding to, and is the j-th legal proposition, and r is a citation It is the number of legal propositions extracted in correspondence with, and is a parameter It is an AI model that has, is a quote , context , a prompt for in-context learning including criteria for extracting legal propositions and one or more examples, and in the above mathematical formula 2, is the number of original text sections generated by dividing the above legal original text, and is 1 or more The k-th original text section among the plurality of original text sections that is subject to the calculation of the basis weight, as an integer below. It is an index that identifies, and It starts from 1 It is a summing index for summing the above multiple original text sections up to, and is a legal proposition Regarding the k-th original text section It is a basis weight indicating the extent to which it can be used as a basis, and is a legal proposition and original text section It is the semantic similarity between, is a metadata consistency based on at least one of a case number, precedent name, court name, date of judgment, statute name, or article number, and is the original text section It is a role score that reflects which part corresponds to the explanation of legal principles, conclusion, judgment, factual circumstances, circumstantial evidence, or additional explanation, and and A legal document citation appropriateness evaluation system that is a weight that adjusts the reflection ratio of semantic similarity, metadata consistency, and role score, classifies the citation as appropriate, caution, or need verification based on the citation appropriateness evaluation result, transmits the citation, the original legal text, and the citation appropriateness evaluation result to the user terminal by matching them with each other, and the user terminal displays the citation, the original legal text, and the citation appropriateness evaluation result received from the server so that they correspond with each other. Claim 11 In claim 10, the AI model comprises: a citation entity extraction model that extracts at least one of a case law name, case number, court name, date of judgment, statute name, article number, and citation from the legal text; a legal context analysis model that derives at least one of a legal claim, requirement, effect, conclusion, and factual determination to be supported by the citation from the context before and after the citation; a text intent analysis model that derives at least one of the intent of a case law ruling, requirement of a statute article, legal effect, time of application, whether it has been amended, and whether it has been repealed from the original text of the law and metadata corresponding to the original text of the law; and a citation appropriateness evaluation model that produces a citation appropriateness evaluation result by comparing the output of the legal context analysis model with the output of the text intent analysis model.
Citation Information
Patent Citations
System and method for evaluating the suitability of reference
KR101033611B1
Legal information evaluation apparatus using ai algorithm and the method thereof
KR1020260038848A
Method for simulating legal cases using artificial intelligence
KR102873606B1
