Multi-Stage Iterative Quality Assurance Method for Document Generation Based on Large Language Model, Apparatus Performing the Same, and Computer-Readable Medium

Abstract

The present invention relates to a method, apparatus, and computer-readable medium for quality assurance of a document using a large-scale language model. The quality assurance method according to the present invention comprises: a hallucination verification step for detecting and correcting hallucination items by comparing factual relationships based on input data for a document generated by a large-scale language model; a cross-verification step for performing cross-verification according to a plurality of predefined verification items; an expert role verification step for verifying by simulating a plurality of expert roles; an enhancement step for improving the document by applying at least one enhancement technique; and an adaptive loop control step for controlling the iterative execution of the quality assurance pipeline, wherein the hallucination verification step, the cross-verification step, the expert role verification step, and the enhancement step are performed sequentially in a fixed order. Accordingly, it is possible to systematically detect hallucination errors in a large-scale language model, improve the consistency and completeness of the document through cross-verification using domain-specific verification items and multi-perspective expert simulation, and ensure the efficiency of quality assurance through adaptive loop control.

Description

Technology Field

[0001] The present invention relates to a multi-stage iterative quality assurance method for ensuring the quality of documents generated by a large-scale language model, an apparatus for performing the same, and a computer-readable medium. More specifically, the present invention relates to a technique for systematically ensuring the quality of document output generated by a large-scale language model by sequentially performing a plurality of quality assurance steps, including hallucination verification, cross-verification, expert role verification, enhancement, and adaptive loop control, in a fixed order. Background Technology

[0002] With the advancement of Large Language Models (LLMs), document generation utilizing LLMs is being attempted in various specialized fields. In particular, as LLMs are being applied to the generation of specialized documents requiring high accuracy and formal rigor, such as patent specifications, legal contracts, and medical reports, the need for technology to systematically guarantee the quality of the generated documents is emerging.

[0003] The Self-Refine approach is known as a prior art. Self-Refine (arXiv:2303.17651) is a general-purpose self-refine framework with a single-loop structure in which an LLM generates feedback on its output and iteratively improves the output by reflecting that feedback. However, because Self-Refine is based on general feedback generation, it has limitations in verifying the structural consistency required by documents in a specific domain. For example, patent specifications contain domain-specific verification items, such as the terms in claims matching the terms used in the description of the invention, all claimed components being supported by embodiments, and reference numerals being consistent throughout the text; however, it is difficult to systematically detect these items using Self-Refine's general-purpose feedback loop. Furthermore, Self-Refine lacks a dedicated detection mechanism for hallucinations that verifies the factual accuracy of the content generated by the LLM by comparing it with input data, and the termination conditions of the iterative loop are unclear, which can lead to quality degradation due to unnecessary iterations or premature convergence.

[0004] Meanwhile, the concept of Dual-Layer Test Guardrails has been proposed in the field of software engineering. Dual-Layer Test Guardrails is an approach that guarantees the output quality of LLM integrated applications by combining a prompt-based function check layer and a scenario-based evaluation layer in a CI / CD (Continuous Integration / Continuous Deployment) pipeline. However, Dual-Layer Test Guardrails are designed to check the format compliance, safety, and reproducibility of LLM output using automated test cases, and are not suitable for verifying the semantic and structural quality of documents generated by LLM based on domain knowledge. Specifically, Dual-Layer Test Guardrails do not include verification items specialized for the formal requirements of specific domain documents (e.g., claim-specification term matching verification in patent specifications, embodiment support verification, etc.), nor do they support loop control mechanisms that adaptively regenerate document content based on verification results.

[0005] As such, in conventional technology, there is no quality assurance pipeline that integrates hallucination verification, domain-specific cross-validation, expert perspective multi-faceted verification, systematic enhancement, and adaptive loop control for LLM-based document generation, and consequently, there is a problem in that it is difficult to systematically guarantee the accuracy, structural consistency, and expertise of the generated documents. Prior art literature

[0006] 1. Madaan, A. et al., “Self-Refine: Iterative Refinement with Self-Feedback,” arXiv:2303.17651, 2023.2. fantasybz, “LLM Dual-Layer Test Guardrails: A Comprehensive Quality Strategy from Prompt Checks to Scenario,” Medium, 2025. The problem to be solved

[0007] The present invention aims to solve the limitations of the aforementioned prior art by providing an integrated quality assurance method that exclusively detects hallucinations in documents generated by LLM to eliminate factual errors, cross-verifies the structural consistency of documents based on multiple domain-specific verification items, multi-facetedly verifies the professional completeness of documents from the perspectives of multiple experts, performs complex advancement including dialectical analysis, first-principle decomposition, and iterative refinement, and adaptively controls the repeated execution of these multiple quality assurance steps.

[0008] In addition, the present invention aims to provide an apparatus for performing the above method and a computer-readable medium storing instructions for executing the above method. means of solving the problem

[0009] To achieve the above objective, according to a first aspect of the present invention, a quality assurance method for a document generated by a large-scale language model is provided, comprising: (a) a hallucination verification step for verifying factual relationships by comparing the content included in the document with input data; (b) a cross-verification step for verifying structural consistency of the document based on a plurality of domain-specific verification items; (c) an expert role verification step for verifying the professional completeness of the document based on a plurality of expert roles; (d) an enhancement step for improving the content by applying at least one enhancement technique to the document; and (e) an adaptive loop control step for determining whether to repeat steps (a) through (d) based on a quality score; wherein steps (a) through (e) are performed sequentially in a fixed order.

[0010] According to a second aspect of the present invention, a quality assurance method is provided, wherein the cross-verification step is performed based on five verification items including term matching verification, embodiment support verification, drawing symbol consistency verification, logical connection verification of problem-solution means-effect, and dependent claim citation consistency verification, the enhancement step applies a plurality of enhancement techniques including dialectical analysis, first-principle decomposition, and iterative refinement, and the adaptive loop control step performs individual activation or deactivation control of each of the steps (a) to (d), early termination when the difference in quality scores between consecutive iterations is less than a predetermined threshold, and iteration limitation according to a predetermined maximum number of iterations.

[0011] According to a third aspect of the present invention, a quality assurance device is provided comprising: a processor; and a memory for storing instructions executed by the processor; wherein the processor is configured to sequentially perform, in a fixed order, a hallucination verification that verifies factual relationships by comparing the contents included in the document with input data, a cross-verification that verifies structural consistency based on five verification items including term matching verification, embodiment support verification, drawing symbol consistency verification, logical connection verification of problem-solution means-effect, and dependent claim citation consistency verification for the document, an expert role verification that verifies the professional completeness of the document based on a plurality of expert roles, an enhancement that improves the content of the document by applying a plurality of enhancement techniques including dialectical analysis, first-principle decomposition, and iterative refinement, and individual activation or deactivation control of each of the hallucination verification, the cross-verification, the expert role verification, and the enhancement, early termination when the difference in quality scores between consecutive iterations is less than a predetermined threshold, and an adaptive loop control that performs iteration limiting according to a predetermined maximum number of iterations.

[0012] According to a fourth aspect of the present invention, a non-transient computer-readable medium is provided that stores instructions that, when executed by a computer, cause the computer to perform a quality assurance method according to the second aspect. Effects of the invention

[0013] According to the present invention, by exclusively performing a comparison of factual relationships with input data in the hallucination verification step for documents generated by an LLM, factual errors that are difficult to detect in conventional general-purpose self-refining methods can be preemptively eliminated.

[0014] Furthermore, according to the present invention, by systematically verifying the structural consistency of a document based on domain-specific verification items including terminology matching, embodiment support, drawing symbol consistency, problem-solution means-effect logical linkage, and dependency citation consistency during the cross-verification step, it is possible to provide domain-level quality assurance that is not available in the prior art.

[0015] In addition, according to the present invention, by simulating multiple expert perspectives, such as examiners and patent attorneys, during the expert role verification stage to verify the professional completeness of the document in a multi-faceted manner, defects that may be overlooked in a single-perspective verification can be detected.

[0016] In addition, according to the present invention, by applying a combination of multiple enhancement techniques including dialectical analysis, first-principle decomposition, and iterative refinement in the enhancement stage, the logical completeness and technical depth of the document can be systematically improved.

[0017] In addition, according to the present invention, by integrally performing individual activation or deactivation control of each quality assurance step, early termination based on the difference in quality scores between consecutive iterations, and a maximum iteration limit in the adaptive loop control step, efficient quality assurance is possible that continues iterations until a sufficient quality level is reached while preventing unnecessary iterations.

[0018] Furthermore, the quality assurance pipeline of the present invention is applicable to the generation of specialized documents in various domains requiring high formal rigor, such as patent specifications as well as legal contracts, thereby providing universal scalability. Brief explanation of the drawing

[0019] FIG. 1 is a block diagram showing the overall configuration of a multi-stage iterative quality assurance pipeline according to one embodiment of the present invention. FIG. 2 is a flowchart showing the processing flow of the hallucination verification step (Q1) according to one embodiment of the present invention. FIG. 3 is a flowchart showing the processing flow of the cross-verification step (Q2) according to one embodiment of the present invention. FIG. 4 is a flowchart showing the processing flow of the expert role verification step (Q3) according to one embodiment of the present invention. FIG. 5 is a flowchart showing the processing flow of an advanced stage (Q4) according to one embodiment of the present invention. FIG. 6 is a flowchart showing the processing flow of an adaptive loop control step (Q5) according to one embodiment of the present invention. FIG. 7 is a flowchart showing the overall sequence of a multi-stage iterative quality assurance method according to one embodiment of the present invention. FIG. 8 is a block diagram showing the hardware configuration of a quality assurance device according to one embodiment of the present invention. Specific details for implementing the invention

[0020] Embodiments of the present invention will be described in detail below with reference to the attached drawings. These embodiments are merely illustrative of the technical concept of the present invention, and the scope of protection of the present invention is not limited to the following embodiments. The scope of protection of the present invention is determined by the matters described in the claims and matters technically inferred therefrom.

[0021] Throughout the specification, the same reference numerals denote the same components. Additionally, if it is determined that a detailed description of known technology in the following description could obscure the essence of the present invention, such detailed description is omitted.

[0022] [Example 1: Multi-stage iterative quality assurance in a patent specification generation pipeline]

[0023] 1. Overall System Configuration

[0024] Referring to FIG. 1, a multi-stage iterative quality assurance system (100) according to one embodiment of the present invention includes a document generation unit (200), a quality assurance pipeline unit (300), and a loop control unit (400).

[0025] The document generation unit (200) generates a draft document (20) based on input data (10) using a large-scale language model (hereinafter 'LLM'). In this embodiment, the input data (10) includes at least one of a patent opinion, an experiment plan, and experiment data, and the draft document (20) includes at least a part of a patent specification.

[0026] The quality assurance pipeline section (300) receives a draft document (20) as input and outputs a verified document (30) by sequentially executing the hallucination verification section (310), the cross-verification section (320), the expert role verification section (330), and the enhancement section (340) in a predetermined fixed order. The fixed order is the sequence of the hallucination verification section (310), the cross-verification section (320), the expert role verification section (330), and the enhancement section (340), and each component receives the output of the preceding component as input. This fixed-order pipeline structure prevents documents containing hallucinations from being propagated to subsequent verification stages and ensures that the verification results of each stage are cumulatively reflected.

[0027] The quality assurance pipeline section (300) may further include a full verification module (350). The full verification module (350) receives all of the LLM outputs produced by the document generation section (200) as verification targets and provides them as input to the quality assurance pipeline section (300). Specifically, the full verification module (350) receives the entire output generated by the LLM in a batch without dividing it into sentence units or section units, and ensures that the entire output passes through the pipeline from the hallucination verification section (310) to the enhancement section (340) without omission. Accordingly, no parts of the LLM output are omitted during verification.

[0028] The loop control unit (400) calculates a quality score for the verified document (30), which is the output of the quality assurance pipeline unit (300), and determines whether to re-execute the quality assurance pipeline unit (300) based on the quality score. The detailed operation of the loop control unit (400) is described in the adaptive loop control step (Q5) described later.

[0029] 2. Hallucination verification step (Q1) - Hallucination verification unit (310)

[0030] Referring to FIG. 2, the hallucination verification unit (310) verifies whether the technical description included in the draft document (20) is supported by the input data (10).

[0031] The hallucination verification unit (310) includes a factual comparison module (311) and a hallucination determination module (312).

[0032] The factual comparison module (311) divides the technical description of the draft document (20) into sentence units or paragraph units and compares each divided unit with the input data (10). Specifically, for each divided unit of the draft document (20), the factual comparison module (311) determines whether there exists a supporting phrase within the input data (10) that supports the technical content of said divided unit. In this embodiment, the determination is performed through LLM-based Natural Language Inference, and one of three labels is assigned to each divided unit: "supported," "not supported," or "uncertain."

[0033] The hallucination verification unit (310) is an LLM used for the above determination and may use a separate verification model different from the generation model used by the document generation unit (200) to generate the draft document (20). The verification model may be a lightweight model with fewer parameters than the generation model. By using a verification model that is lower cost than the generation model, verification from an independent perspective is possible while reducing the computational cost and processing time required for hallucination verification. However, the specific architecture and parameter scale of the verification model are not limited to this, and it is also possible to use a model identical to the generation model.

[0034] The hallucination determination module (312) receives the output of the factual comparison module (311) and classifies the partition unit labeled "unsupported" as a suspected hallucination item. The hallucination determination module (312) generates a correction instruction for the suspected hallucination item, the correction instruction including deleting the item or rewriting it based on the input data (10). For the partition unit labeled "unidentifiable," the hallucination determination module (312) adds the item to a separate review list and sets a flag requesting confirmation from the user.

[0035] The hallucination verification unit (310) outputs an intermediate document (21) reflecting corrections to suspected hallucination items and transmits it to the cross-verification unit (320). At this time, a hallucination verification result report is attached to the intermediate document (21) as metadata, and the report includes the number of detected hallucination items, the number of correction items, and the number of unidentifiable items.

[0036] 3. Cross-validation step (Q2) - Cross-validation unit (320)

[0037] Referring to FIG. 3, the cross-verification unit (320) receives an intermediate document (21) and performs cross-verification on a plurality of predetermined verification items.

[0038] In this embodiment, the plurality of verification items are the following five items specialized in the patent specification (hereinafter '5-item verification').

[0039] The first verification item (E1) is a term matching verification. The cross-verification unit (320) compares the terms used in the claims with the terms used in the description of the invention to detect whether different terms are used for the same concept. Specifically, the cross-verification unit (320) refers to a pre-registered term correspondence table (321) to determine whether specification terms corresponding to each term in the claims are consistently used in the description of the invention. The term correspondence table (321) is a data structure that stores the correspondence relationships between claim terms, specification terms, and input data terms; it is created when the specification is first written and is updated whenever a new term is discovered during the verification stage. The term correspondence table (321) is stored in the database of the storage unit (540) described later, and the cross-verification unit (320) refers to the latest state of the term correspondence table (321) through a query request to the database.

[0040] The second verification item (E2) is a verification supporting the embodiment. The cross-verification unit (320) verifies whether each component of the claim is specifically described in the specific details for implementing the invention (hereinafter referred to as 'embodiments'). Specifically, it extracts each component of the claim and determines whether there is a technical description corresponding to said component in the body of the embodiment. If a component for which no corresponding technical description exists is detected, the cross-verification unit (320) classifies it as a risk of insufficient description (Article 42, Paragraph 3 of the Patent Act) and generates a supplementary instruction.

[0041] The third verification item (E3) is a verification of drawing symbol consistency. The cross-verification unit (320) compares the drawing symbols used in the text of the embodiment with the symbols described in the brief description of the drawing and the description of the symbols to detect discrepancies in symbol numbers, discrepancies in names, or the use of unregistered symbols.

[0042] The fourth verification item (E4) is a verification of the logical connection between the task, the means of solution, and the effect. The cross-verification unit (320) logically tracks whether each task described in the task to be solved is solved by the corresponding configuration of the means of solution of the task, and whether the effect described in the effect of the invention is derived by said solution. If a task or effect with a disconnected logical connection is detected, the cross-verification unit (320) generates a supplementary instruction for the corresponding item.

[0043] The fifth verification item (E5) is a verification of the consistency of dependent claim citations. The cross-verification unit (320) verifies whether there exists an independent claim or a parent dependent claim that each dependent claim cites, whether the citation relationship is circular, and whether the component defined by the dependent claim is actually included in the citation claim.

[0044] The cross-verification unit (320) aggregates the results of the five-item verification by item and generates a cross-verification result report in which a judgment of "pass," "fail," or "warning" is assigned to each item. The report includes specific locations (claim number, paragraph number, code number, etc.) and supplementary instructions for the non-compliant items.

[0045] The cross-verification unit (320) outputs an intermediate document (22) with corrections reflected for items judged as unsuitable and transmits it to the expert role verification unit (330).

[0046] 4. Expert Role Verification Step (Q3) - Expert Role Verification Department (330)

[0047] Referring to FIG. 4, the expert role verification unit (330) receives an intermediate document (22) and verifies the quality of the document from the perspective of multiple experts.

[0048] The expert role verification unit (330) generates verification opinions from each expert's perspective by applying multiple expert role prompts to the LLM sequentially or in parallel. In this embodiment, the multiple expert roles include an examiner role (331) and a patent attorney role (332).

[0049] The examiner role (331) is for the LLM to review the intermediate document (22) by simulating the perspective of a patent examiner. Specifically, the examiner role (331) reviews each independent claim for whether there is a violation of novelty (Article 29, Paragraph 1 of the Patent Act), a violation of inventive step (Article 29, Paragraph 2 of the Patent Act), and whether there is a lack of description (Article 42, Paragraphs 3 and 4 of the Patent Act), and structures the review results into a notice of submission of opinion and outputs them. At this time, the examiner role (331) analyzes the differences in the components of each independent claim compared to the prior art by referring to the list of prior art documents included in the input data (10).

[0050] The patent attorney role (332) is for the LLM to simulate the perspective of the applicant's patent attorney and generate a response strategy to the results of the examination by the examiner role (331). Specifically, for each reason for rejection pointed out by the examiner role (331), the patent attorney role (332) proposes an amendment plan (adding a limitation to the claim), an opinion argument plan (arguing for differences from prior art), or a dependent claim replacement plan (reducing the scope of rights to dependent claims when independent claims are rejected).

[0051] The expert role verification unit (330) synthesizes the verification results of the examiner role (331) and the patent attorney role (332) and generates correction instructions for items that require supplementation. An intermediate document (23) reflecting the correction instructions is transmitted to the enhancement unit (340). An expert verification result report is attached to the intermediate document (23) as metadata, and the report includes points raised from the examiner's perspective, countermeasures from the patent attorney's perspective, and the final reflection results.

[0052] 5. Advanced stage (Q4) - Advanced unit (340)

[0053] Referring to FIG. 5, the enhancement unit (340) receives an intermediate document (23) and improves the technical quality of the document by applying three techniques: dialectical analysis, first-principle decomposition, and iterative refinement.

[0054] 5-1. Dialectical Analysis - Dialectical Analysis Module (341)

[0055] The dialectical analysis module (341) performs a thesis-antithesis-synthesis dialectical cycle for the claim strategy or technical interpretation of the intermediate document (23). Specifically, the dialectical analysis module (341) performs the following processing.

[0056] In the thesis stage, the claim strategy or technical interpretation adopted in the current document is established as a proposition. In the antithesis stage, counterarguments to the said proposition are generated. These counterarguments include similarity to prior art, excessive expansion of the scope of rights, or the possibility of insufficient disclosure. In the synthesis stage, the proposition and counterarguments are combined to derive an improved strategy or interpretation.

[0057] The dialectical analysis module (341) repeats the thesis-antithesis-synthesis cycle within a predetermined maximum number of repetitions (N times). The synthesis derived from each repetition is used as the thesis for the next repetition. The dialectical analysis module (341) determines that the two repetitions have converged and terminates the repetition if the sums derived from two consecutive repetitions are substantially identical. The substantial identity corresponds to the case where the similarity is greater than or equal to a predetermined convergence threshold when comparing the texts of the two sums using LLM-based semantic similarity analysis. In one embodiment, the semantic similarity analysis may involve converting the texts of the two sums into embedding vectors, calculating the cosine similarity between the two embedding vectors, and comparing it with the convergence threshold.

[0058] 5-2. First-principle decomposition - First-principle decomposition module (342)

[0059] The first principle decomposition module (342) decomposes and analyzes the technical configuration of the invention described in the intermediate document (23) to the level of first principles. Specifically, the first principle decomposition module (342) performs the following five-step processing.

[0060] Step 1 (Assumption Analysis): Extract the technical assumptions assumed by the current specification.

[0061] Step 2 (Decomposition): The above-mentioned extracted premise is decomposed into the smallest technical units.

[0062] Step 3 (Assumption Removal): Identify assumptions among the decomposed units that are not verified by the input data (10).

[0063] Step 4 (Separation of Core Principles): After the assumptions are eliminated, select the core principles corresponding to the essence of the invention from the remaining technical principles.

[0064] Step 5 (Redesign): Based on the above core principles, derive methods to improve the specification structure or technical description.

[0065] The first principle decomposition module (342) generates correction instructions for descriptions containing unfounded assumptions, or supplementary instructions if core principles are insufficiently described in the specification, based on the analysis results of the five steps above.

[0066] 5-3. Iterative Refining - Iterative Refining Module (343)

[0067] The iterative refinement module (343) specifically modifies the text of the intermediate document (23) based on the output of the dialectical analysis module (341) and the first principle decomposition module (342). The iterative refinement module (343) sorts the items to be modified according to priority, sorting modifications related to incomplete descriptions as the highest priority, modifications related to strengthening inventiveness as the next highest priority, and readability improvement as the lowest priority.

[0068] The enhancement unit (340) outputs an enhanced document (24) in which dialectical analysis, first-principle decomposition, and iterative refinement are all completed. An enhancement result report is attached to the enhanced document (24) as metadata, and said report includes the number of dialectical cycles, whether convergence has occurred, a list of baseless assumptions identified in the first-principle analysis, and a list of iterative refinement correction items.

[0069] 6. Adaptive loop control step (Q5) - Loop control unit (400)

[0070] Referring to FIG. 6, the loop control unit (400) receives an enhancement document (24), which is the output of the enhancement unit (340), and determines whether to re-execute the entire quality assurance pipeline unit (300).

[0071] The loop control unit (400) includes a step-by-step activation control module (410), a quality evaluation module (420), and a termination determination module (430).

[0072] 6-1. Step-by-step activation control - Step-by-step activation control module (410)

[0073] The step-by-step activation control module (410) individually sets the activation (ON) or deactivation (OFF) state for each of the hallucination verification unit (310), cross-verification unit (320), expert role verification unit (330), and advancement unit (340) included in the quality assurance pipeline unit (300).

[0074] The step activation control module (410) is configured to disable steps in which no non-compliant items were detected by referring to the verification result report of each step in the previous loop iteration. For example, if the verification result of the hallucination verification unit (310) in the previous loop iteration is "supported" for all items, the hallucination verification unit (310) is disabled in the next loop iteration to prevent an increase in computational cost due to unnecessary re-verification. The disabled steps pass-through the input as is to the next step.

[0075] 6-2. Quality Evaluation - Quality Evaluation Module (420)

[0076] The quality evaluation module (420) calculates a quality score for the advanced document (24). The quality score is a numeric value between 0 and 100, and is calculated by combining the hallucination verification result, the cross-verification 5-item result, the expert role verification result, and the advanced result according to predetermined weights. In this embodiment, the weights are assigned as 0.3 for the hallucination verification result, 0.3 for the cross-verification 5-item result, 0.2 for the expert role verification result, and 0.2 for the advanced result, but are not limited thereto.

[0077] In one embodiment, the verification result of each step is quantified by item, with a score of 1.0 for pass, 0.5 for warning, and 0.0 for fail, and the average value of the scores for each item within each step can be calculated as the score for that step.

[0078] 6-3. Termination Determination - Termination Determination Module (430)

[0079] The termination determination module (430) determines whether to terminate the re-execution of the quality assurance pipeline section (300) based on the quality score calculated by the quality evaluation module (420). The termination determination module (430) terminates the re-execution if at least one of the following three termination conditions is satisfied.

[0080] The first termination condition is when the quality score of the current loop iteration is greater than or equal to a predetermined quality threshold. The second termination condition is when the difference between the quality score of the current loop iteration and the quality score of the previous loop iteration is less than a predetermined early termination threshold. This indicates that quality improvement has converged and there is no benefit in further iterations. The third termination condition is when the number of loop iterations reaches a predetermined maximum number of iterations.

[0081] When the termination determination module (430) determines that the re-execution is terminated, the loop control unit (400) outputs the advanced document (24) as a verified document (30). When the termination determination module (430) determines that the re-execution is continued, the loop control unit (400) reintroduces the advanced document (24) into the hallucination verification unit (310) of the quality assurance pipeline unit (300), and applies the activation state updated by the step-by-step activation control module (410).

[0082] 7. Overall Processing Flow

[0083] Referring to FIG. 7, a multi-stage iterative quality assurance method according to one embodiment of the present invention includes the following steps.

[0084] In the document generation step (S100), a draft document (20) based on input data (10) is generated using a large-scale language model.

[0085] In the hallucination verification step (S200), the technical description of the draft document (20) is compared with the input data (10) to verify whether it is hallucinatory, and an intermediate document (21) is generated that reflects corrections to items suspected of being hallucinatory.

[0086] In the cross-verification step (S300), five items of verification are performed on the intermediate document (21), including term matching (E1), support for embodiments (E2), consistency of drawing symbols (E3), logical linkage of the problem-solution means-effect (E4), and consistency of dependent claim citations (E5), and an intermediate document (22) is generated with corrections for non-conforming items reflected.

[0087] In the expert role verification step (S400), verification is performed on the intermediate document (22) from the examiner's perspective and the patent attorney's perspective, and an intermediate document (23) with supplements reflected is generated.

[0088] In the enhancement stage (S500), dialectical analysis, first-principle decomposition, and iterative refinement are performed on the intermediate document (23) to generate an enhanced document (24).

[0089] In the adaptive loop control step (S600), the quality score of the advanced document (24) is calculated, and whether the termination condition is satisfied is determined. If the termination condition is not satisfied, the process is restarted from the hallucination verification step (S200), but steps in which no non-conforming items were detected in the previous iteration are disabled. If the termination condition is satisfied, the advanced document (24) is output as a verified document (30).

[0090] 8. Device Configuration

[0091] Referring to FIG. 8, a quality assurance device (500) according to one embodiment of the present invention includes a processor (510) and a memory (520) that stores instructions executed by the processor (510).

[0092] The memory (520) includes a non-transient computer-readable medium for storing the above instructions, and when the above instructions are executed by the processor (510), the processor (510) causes the hallucination verification step (S200), cross-verification step (S300), expert role verification step (S400), enhancement step (S500), and adaptive loop control step (S600) to perform the above hallucination verification step (S200), cross-verification step (S300), expert role verification step (S400), enhancement step (S500), and adaptive loop control step (S600).

[0093] The quality assurance device (500) may further include a communication interface (530). The communication interface (530) may be configured to connect to an external large-scale language model service to transmit inference requests and receive inference results. In this case, the processor (510) requests LLM inference through the communication interface (530) and receives the results to perform subsequent processing in the operation of each of the hallucination verification unit (310), cross-verification unit (320), expert role verification unit (330), enhancement unit (340), and loop control unit (400).

[0094] The quality assurance device (500) may further include a storage unit (540). The storage unit (540) stores a term correspondence table (321), a verification result report for each step, a loop iteration history (including the quality score, activation status, and termination judgment result for each iteration), and a final verification completion document (30).

[0095] The embodiments described above are intended merely to specifically explain the technical concept of the present invention and should not be interpreted as limiting the scope of protection of the present invention. The scope of protection of the present invention is determined by the matters described in the claims and matters technically inferred therefrom. It is obvious that a person skilled in the art to which the present invention pertains can make various modifications and variations within the scope of the technical concept of the present invention without departing from it.

[0096] [Example 2: Application of Legal Contract Domain]

[0097] The following describes an embodiment in which the multi-stage iterative quality assurance method according to the present invention is applied to the domain of legal contracts. In Example 1, a patent specification was used as the target document; however, the quality assurance pipeline of the present invention is not limited to a specific domain and can be universally applied by modifying the verification items of each stage according to the domain of the target document. In this embodiment, the application method of each stage is explained using a legal contract, specifically a commercial contract, as the target document.

[0098] [Input Data Configuration]

[0099] In this embodiment, the input data (10) includes a summary of contract terms, party information, transaction terms specifications, and information on the designation of governing law. The document generation unit (200) receives the input data (10) and generates a draft contract document (20) through a large-scale language model.

[0100] In one embodiment, the input data (10) may further include contract samples included in CUAD (Contract Understanding Atticus Dataset) as reference data. CUAD is a public dataset containing annotations for 41 key clause types for more than 500 actual corporate contracts, and the document generation unit (200) can utilize the clause type classification system of CUAD as a clause structure template when generating a draft contract. Additionally, the annotation data of CUAD can be utilized as verification standard data in the cross-validation step described later.

[0101] [Hallucination Verification Phase (Q1) - Phase S200]

[0102] The hallucination verification unit (310) of the quality assurance pipeline unit (300) performs a hallucination verification step on the draft document (20). The hallucination in the legal contract domain is different in type from the hallucination in the patent specification domain of Example 1.

[0103] The factual comparison module (311) compares the factual information recorded in the draft document (20) with the input data (10). Specifically, the factual comparison module (311) verifies whether (i) identification information such as the names, addresses, and registration numbers of the parties matches the input data (10), (ii) transaction conditions such as the transaction subject, quantity, unit price, and total amount match the figures recorded in the input data (10), and (iii) date information such as the contract period, delivery schedule, and performance deadline matches the input data (10).

[0104] The hallucination determination module (312) classifies inconsistency items as suspected hallucination items based on the comparison results of the factual comparison module (311). Since numerical hallucinations in the legal contract domain are directly linked to financial loss, the hallucination determination module (312) may assign a higher severity grade to inconsistencies regarding amount, quantity, and date than to general hallucinations in Example 1. The hallucination determination module (312) corrects the suspected hallucination items based on input data (10) and outputs an intermediate document (21).

[0105] [Cross-validation Step (Q2) - Step S300]

[0106] The cross-verification unit (320) performs a cross-verification step on the intermediate document (21). In this embodiment, the cross-verification unit (320) applies five verification items (C1 to C5) specialized for the legal contract domain.

[0107] (C1) Verification of term match between definition clause and text: The cross-verification unit (320) extracts a list of terms defined in the definition clause of the contract and compares the usage status of the corresponding terms throughout the text of the contract. Errors are detected if a term defined in the definition clause is used in the text with a meaning different from the definition or is replaced with a different expression, if there is a technical term used in the text but not defined in the definition clause, and if there is a term defined in the definition clause but never used in the text. This corresponds to item E1 of Example 1.

[0108] (C2) Verification of duty-right clause balance: The cross-verification unit (320) extracts the duty clauses and right clauses of each party listed in the contract and verifies whether there is a right corresponding to each duty and whether there is an duty corresponding to each right. It detects errors in cases where an obligation is imposed on one party but no remedy for non-performance of the obligation is specified for the other party, where a right is granted to one party but no conditions or limitations on the exercise of the right are specified, and where conflicting obligations regarding the same matter are listed in multiple clauses. This corresponds to item E2 of Example 1.

[0109] (C3) Verification of consistency of clause number cross-references: The cross-verification unit (320) extracts all cross-references that refer to other clauses within the body of the contract and verifies whether the referenced clause number matches an actual existing clause. It detects errors in cases where there is no clause corresponding to the referenced clause number, where the content of the referenced clause is irrelevant to the reference context, and where the existing cross-reference is invalidated by the insertion, deletion, or change of order of the clause. This corresponds to item E3 of Example 1.

[0110] (C4) Verification of condition-effect logical connection: The cross-verification unit (320) verifies whether the logical connection of condition precedent, obligation, and legal consequence is complete for the conditional clauses stated in the contract. Errors are detected in cases where a condition precedent is specified but the legal consequence upon fulfillment of the condition is not stated, where a legal consequence is stated but the condition precedent triggering it is unclear, and where an intermediate step of the condition-effect chain is omitted and a logical leap exists. This corresponds to item E4 of Example 1.

[0111] (C5) Verification of consistency between governing law and dispute resolution clause: The cross-verification unit (320) verifies the consistency between the governing law and the dispute resolution mechanism (arbitration or litigation) specified in the contract. Errors are detected if a dispute resolution method that is not permitted in the legal system of the country designated as the governing law is specified, if the arbitration institution specified in the arbitration clause is an institution not recognized in the country of the governing law, if multiple governing laws are specified in conflict, and if the dispute resolution clause is completely omitted. This corresponds to item E5 of Example 1.

[0112] The cross-verification unit (320) synthesizes the verification results of the above C1 to C5 items and outputs an intermediate document (22).

[0113] [Expert Role Verification Phase (Q3) - Phase S400]

[0114] The expert role verification unit (330) performs an expert role verification step on the intermediate document (22). In this embodiment, the expert role verification unit (330) simulates an expert role suitable for the legal contract domain.

[0115] In Example 1, the expert role verification unit (330) simulated the examiner role (331) and the patent attorney role (332), but in this example, the expert role is modified according to the domain. Specifically, the opposing attorney role is simulated in correspondence with the examiner role (331), and the advisory attorney role is simulated in correspondence with the patent attorney role (332).

[0116] The role of the opposing attorney reviews the clauses of the contract from a perspective favorable to the opposing party to identify clauses disadvantageous to their own party, interpretation risks arising from ambiguous wording, and missing protection clauses. The role of the counsel proposes response strategies for the risks identified by the opposing attorney, through clause amendments, the insertion of additional provisions, or clarification of wording.

[0117] The expert role verification unit (330) outputs an intermediate document (23) by reflecting the review-correspondence results between the two roles.

[0118] [Advanced Stage (Q4) - Stage S500]

[0119] The enhancement unit (340) performs an enhancement step on the intermediate document (23). The enhancement unit (340) includes a dialectical analysis module (341), a first principle decomposition module (342), and a repetitive refinement module (343), just as in Example 1.

[0120] (a) Dialectical analysis: The dialectical analysis module (341) performs a thesis-antithesis-synthesis cycle on key issues of the contract. For example, regarding the damages limit clause, a proposition (thesis) is established as “the damages limit is set to 100% of the contract price,” an antithesis (antithesis) is established as “setting the limit may induce moral hazard by exempting damages caused by gross negligence or intent,” and a synthesis is derived as “the limit of 100% of the contract price is applied to general damages, but damages caused by gross negligence or intent are excluded from the application of the limit.”

[0121] (b) First principle decomposition: The first principle decomposition module (342) performs a five-step decomposition on the core clauses of the contract. For example, regarding the intellectual property attribution clause, (step 1) the premise that "the intellectual property rights of all development results performed by the trustee belong to the trustor" is extracted as a premise analysis, (step 2) the scope of "development results," the type of "intellectual property rights," and the timing of "attribution" are decomposed as a decomposition, (step 3) the assumption elimination identifies the baseless assumption that "the trustee's existing technology (background IP) is also included in the subject of attribution?", (step 4) the essence that "the attribution of new development outputs and the license of existing technology must be distinguished" is derived as a core principle separation, and (step 5) the improved clause structure is derived as a redesign.

[0122] (c) Iterative Refinement: The iterative refinement module (343) modifies the contract text sequentially, starting with the highest priority items, based on the output of the dialectical analysis module (341) and the first principle decomposition module (342). Priority is determined based on the severity of legal risk, and clauses with a high probability of financial loss have a higher priority than wording clarification.

[0123] [Adaptive Loop Control Step (Q5) - Step S600]

[0124] The loop control unit (400) performs an adaptive loop control step for the advanced document (24). The structure and operation of the loop control unit (400) are the same as in Example 1.

[0125] The step-by-step activation control module (410) can set the activation or deactivation of each step according to the characteristics of the legal contract domain. For example, if the input data (10) includes contract conditions that have already been reviewed by a legal expert, the expert role verification step (Q3) can be deactivated and only the remaining steps can be executed.

[0126] The quality evaluation module (420) calculates a quality score for the advanced document (24). The quality score in the legal contract domain can be calculated based on the number of remaining errors in each of the above C1 to C5 items, the number of unresolved risks identified in the expert role verification, and the advanced reflection rate.

[0127] The termination determination module (430) terminates the loop when at least one of the three termination conditions identical to those in Example 1 is satisfied, namely, (i) when the quality score is greater than or equal to the target threshold, (ii) when the difference between the quality score of the current round and the quality score of the previous round is less than a predetermined threshold (early termination), and (iii) when the number of repetitions reaches a predetermined maximum number.

[0128] [Correspondence between Example 1 and Example 2]

[0129] As explained above, the multi-stage iterative quality assurance method according to the present invention can be applied to various document domains by adaptively modifying the verification items of each stage according to the domain of the target document.

[0130] Q1 In the hallucination verification step, Example 1 compares the facts against the input data (opinion statement, experiment plan, experiment data), whereas Example 2 compares the facts against the party information, transaction conditions, and date information. Both examples share the basic structure of the factual comparison module (311) and the hallucination determination module (312), but the comparison target items differ depending on the domain.

[0131] In the Q2 cross-verification step, the five verification items (E1 to E5) of Example 1 correspond to the five verification items (C1 to C5) of Example 2, respectively. Specifically, E1 (term matching based on the term matching table) corresponds to C1 (definition clause-text term matching), E2 (verification supporting the example) corresponds to C2 (balance of obligation-right clauses), E3 (consistency of drawing symbols) corresponds to C3 (consistency of clause number cross-references), E4 (logic linkage of task-solution means-effect) corresponds to C4 (logic linkage of condition-effect), and E5 (consistency of dependent claim citation) corresponds to C5 (consistency of governing law-dispute resolution).

[0132] In the Q3 expert role verification step, the examiner role (331) and patent attorney role (332) of Example 1 are transformed into the opposing attorney role and the advisory attorney role, respectively, in Example 2. Both examples share a pair structure of adversarial review roles and response strategy roles, but the review perspective of each role varies according to the stakeholder structure of the domain.

[0133] In the Q4 advanced stage, the basic structure of the dialectical analysis module (341), the first principle decomposition module (342), and the iterative refinement module (343) is the same regardless of the domain. In Example 2, the subject of the dialectical analysis is transformed into an issue of a contract clause, and the subject of the first principle decomposition is transformed into a premise of a contract clause, but the processing procedure of the thesis-antithesis-synthesis cycle and the five-step decomposition itself is the same as in Example 1.

[0134] In the Q5 adaptive loop control step, the three termination conditions (goal achievement, early termination, number of times limit) and the step-by-step activation / deactivation structure of the loop control unit (400) are the same as in Example 1 and Example 2. Only the criteria for calculating the quality score are modified according to the domain, and in the legal contract domain, the number of remaining errors and unresolved risks of items C1 to C5 serve as the basis for calculating the quality score.

[0135] As confirmed by the correspondence above, the basic structure of the 5-step fixed-order pipeline (Q1→Q2→Q3→Q4→Q5) of the present invention is maintained regardless of the domain, and the detailed items of Q2 cross-validation and the specific personas of the Q3 expert role are modified according to the domain. Accordingly, the "document" of claim 1 is not limited to a patent specification and is applicable to various types of professional documents, including legal contracts.

[0136] In addition to the above Examples 1 and 2, the present invention can be applied to specialized document areas where domain-specific verification items exist, such as pharmaceutical clinical trial reports, financial regulatory compliance reports, and technical standard documents. Q2 verification items in each domain are defined according to the document quality standards of the domain, and Q3 expert roles are set according to the stakeholder structure of the domain. Industrial applicability

[0137] The multi-stage iterative quality assurance method, apparatus, and computer-readable medium according to the present invention can be used in industrial fields that generate documents using a large-scale language model.

[0138] Specifically, the present invention can be applied to document generation services, intellectual property management systems, legal document automation systems, etc., by providing a quality assurance pipeline including hallucination verification, domain-specific cross-validation, expert role verification, enhancement, and adaptive loop control in the process of generating specialized documents requiring high accuracy and formal rigor, such as patent specifications and legal contracts.

[0139] Furthermore, since the present invention can be implemented in software on a general-purpose computing device including a processor and memory and can be implemented in a form that interacts with an external large-scale language model service through a communication interface, it can be industrially utilized in various computing environments, including cloud-based services and on-premises environments. Explanation of the symbols

[0140] 10: Input data 20: Draft Document 21: Intermediate document (Q1 output) 22: Intermediate document (Q2 output) 23: Intermediate document (Q3 output) 24: Advanced Document (Q4 Output) 30: Verified Document 100: Multi-stage iterative quality assurance system 200: Document Creation Section 300: Quality Assurance Pipeline Division 310: Hallucination Verification Department 311: Fact-checking Module 312: Hallucination Judgment Module 320: Cross-validation section 321: Terminology Correspondence Table 330: Expert Role Verification Department 331: Examiner Role 332: Role of a Patent Attorney 340: High-level Fire Department 341: Dialectical Analysis Module 342: First Principles Disassembly Module 343: Iterative Refinement Module 350: Full Verification Module 400: Loop control unit 410: Step-by-step activation control module 420: Quality Evaluation Module 430: Termination Determination Module 500: Quality Assurance Device 510: Processor 520: Memory 530: Communication interface 540: Storage section S100: Document creation step S200: Hallucination Verification Stage S300: Cross-validation step S400: Expert Role Verification Stage S500: Advanced stage S600: Adaptive loop control step

Claims

Claim 1 A method for ensuring the quality of a document using a large-scale language model, comprising: a hallucination verification step for detecting and correcting hallucination items by comparing factual relationships based on input data for a document generated by the large-scale language model; a cross-verification step for performing cross-verification according to a plurality of predefined verification items for a document that has undergone the hallucination verification step; an expert role verification step for verifying a plurality of expert roles by simulating them for a document that has undergone the cross-verification step; an enhancement step for improving a document by applying at least one enhancement technique for a document that has undergone the expert role verification step; and an adaptive loop control step for controlling the iterative execution of a quality assurance pipeline including the hallucination verification step, the cross-verification step, the expert role verification step, and the enhancement step; wherein the hallucination verification step, the cross-verification step, the expert role verification step, and the enhancement step are performed sequentially in a fixed order. Claim 2 A method for quality assurance of a document according to claim 1, wherein the hallucination verification step comprises: a step of classifying inconsistent items as suspected hallucination items by comparing factual information included in the input data and content described in the document item by item; and a step of correcting the suspected hallucination items based on the input data. Claim 3 A method for quality assurance of a document according to claim 1, wherein the cross-verification step comprises the step of extracting the corresponding description of the document for each of five predefined verification items and determining whether the description satisfies the criteria of the verification item. Claim 4 A method for quality assurance of a document according to paragraph 3, wherein at least one of the five verification items includes a term matching verification item, and the verification of the term matching verification item is to determine whether a term in the document matches a term registered in the term matching table by referring to a pre-stored term matching table. Claim 5 A method for ensuring the quality of a document, wherein the expert role verification step comprises: an examiner role simulation for deriving grounds for possible rejection from an examiner's perspective regarding the document; and a patent attorney role simulation for establishing a response strategy to the grounds for possible rejection. Claim 6 A method for ensuring the quality of a document, wherein, in claim 1, the enhancement step comprises the step of performing a dialectical analysis on at least a portion of the document to derive an antithesis to a proposition and generate a synthesis. Claim 7 A method for ensuring the quality of a document according to claim 6, wherein the above dialectical analysis repeats the thesis-antithesis-synthesis cycle up to N times (N is a natural number greater than or equal to 2), and terminates the repetition by determining convergence when the sum generated in two consecutive cycles is the same. Claim 8 A method for ensuring the quality of a document according to claim 1, wherein the enhancement step comprises: a step of performing first-principle decomposition on at least a portion of the document to decompose premises into minimum units, identifying and removing unfounded assumptions, and then reconstructing the corresponding portion based on core principles. Claim 9 A method for ensuring the quality of a document according to claim 1, wherein the enhancement step comprises: a step of performing a dialectical analysis on at least a portion of the document; a step of performing a first-principle decomposition on at least a portion of the document; and a step of repeatedly refining the document by reflecting the results of the dialectical analysis and the first-principle decomposition. Claim 10 A method for quality assurance of a document according to claim 1, wherein the adaptive loop control step comprises: a step of calculating a quality score of the document in each iteration of the quality assurance pipeline; and an early termination step of terminating the iteration execution if the difference between the quality score of the current iteration and the quality score of the previous iteration is less than a predetermined threshold. Claim 11 A method for quality assurance of a document according to claim 10, wherein the above-mentioned early termination step terminates the above-mentioned iterative execution when the value obtained by subtracting the quality score of the (N-1)th iteration from the quality score of the Nth iteration (N is a natural number greater than or equal to 2) is less than the above-mentioned threshold. Claim 12 A method for quality assurance of a document according to claim 1, wherein the adaptive loop control step comprises a step of individually setting activation or deactivation for each of the hallucination verification step, the cross-verification step, the expert role verification step, and the enhancement step; wherein the deactivated steps are skipped during the execution of the quality assurance pipeline, while maintaining a fixed order between the activated steps. Claim 13 A method for quality assurance of a document according to claim 1, wherein the document is a patent specification and the input data includes at least one of a patent opinion, an experimental plan, and experimental data. Claim 14 A method for guaranteeing the quality of a document using a large-scale language model comprises: a hallucination verification step for a document generated by the large-scale language model, wherein the hallucination items are detected and corrected by comparing factual relationships based on input data; a cross-verification step for a document that has undergone the hallucination verification step, wherein cross-verification is performed according to five predefined verification items, and wherein the five verification items include term matching verification, embodiment support verification, drawing symbol consistency verification, logical connection verification of the problem, solution means, and effect, and dependent claim citation consistency verification; an expert role verification step for a document that has undergone the cross-verification step, wherein multiple expert roles are simulated and verified; and an enhancement step for a document that has undergone the expert role verification step, wherein the document is improved by applying an enhancement technique including dialectical analysis, first-principle decomposition, and iterative refinement. A method for quality assurance of a document, comprising: an adaptive loop control step that controls the iterative execution of a quality assurance pipeline including the hallucination verification step, the cross-verification step, the expert role verification step, and the enhancement step, wherein the adaptive loop control step includes individual activation or deactivation settings for each step, early termination based on the improvement of the quality score, and a limit on the maximum number of iterations; wherein the hallucination verification step, the cross-verification step, the expert role verification step, and the enhancement step are performed sequentially in a fixed order. Claim 15 A device for guaranteeing the quality of a document using a large-scale language model comprises: a processor; and a memory for storing instructions executed by the processor; wherein the processor comprises: a hallucination verification unit that detects and corrects hallucination items by comparing factual relationships based on input data for a document generated by the large-scale language model; a cross-verification unit that performs cross-verification according to five predefined verification items for a document that has passed through the hallucination verification unit, wherein the five verification items include term matching verification, embodiment support verification, drawing symbol consistency verification, logical connection verification of a problem, solution means, and effect, and dependent claim citation consistency verification; an expert role verification unit that verifies a plurality of expert roles by simulating them for a document that has passed through the cross-verification unit; and an enhancement unit that improves a document by applying an enhancement technique including dialectical analysis, first-principle decomposition, and iterative refinement for a document that has passed through the expert role verification unit. A document quality assurance device configured to function as a loop control unit that controls the iterative execution of a quality assurance pipeline including the hallucination verification unit, the cross-verification unit, the expert role verification unit, and the enhancement unit, wherein the loop control unit includes individual activation or deactivation settings for each stage, early termination based on the improvement range of the quality score, and a limit on the maximum number of iterations, wherein the hallucination verification unit, the cross-verification unit, the expert role verification unit, and the enhancement unit are executed sequentially in a fixed order. Claim 16 In item 15, the hallucination verification unit performs detection of the hallucination item using a verification model different from the large-scale language model, wherein the verification model is a low-cost model that uses fewer computational resources than the large-scale language model, a document quality assurance device. Claim 17 In item 15, the cross-verification unit performs term matching verification by linking with a term correspondence table database stored in a storage unit, a document quality assurance device. Claim 18 A document quality assurance device according to claim 15, wherein the enhancement unit is configured to repeat the thesis-antithesis-synthesis cycle in the dialectical analysis up to N times (N is a natural number greater than or equal to 2), and to terminate the repetition by determining convergence if the sum generated in two consecutive cycles is the same. Claim 19 A document quality assurance device according to claim 15, wherein the loop control unit is configured to record the execution history of each of the hallucination verification unit, the cross-verification unit, the expert role verification unit, and the enhancement unit step by step, and to determine whether to activate or deactivate each step based on the execution history. Claim 20 In paragraph 15, the processor is configured to function further as a full verification module that receives and verifies the output of the large-scale language model in a full batch, and the full verification module inputs the output of the large-scale language model into the quality assurance pipeline after the output is completed, a document quality assurance device. Claim 21 When executed by a computer, the computer performs a hallucination verification step in which, for a document generated by a large-scale language model, factual relationships are compared based on input data to detect and correct hallucination items; for a document that has undergone the hallucination verification step, cross-verification is performed according to five predefined verification items, wherein the five verification items include term matching verification, embodiment support verification, drawing symbol consistency verification, logical connection verification of the problem, solution means, and effect, and dependent claim citation consistency verification; for a document that has undergone the cross-verification step, an expert role verification step in which multiple expert roles are simulated and verified; and for a document that has undergone the expert role verification step, an enhancement step in which an enhancement technique including dialectical analysis, first-principle decomposition, and iterative refinement is applied to improve the document. A non-transient computer-readable medium that stores instructions for controlling the iterative execution of a quality assurance pipeline including the hallucination verification step, the cross-verification step, the expert role verification step, and the enhancement step, wherein the adaptive loop control step includes individual activation or deactivation settings for each step, early termination based on the improvement of the quality score, and a limit on the maximum number of iterations; wherein the hallucination verification step, the cross-verification step, the expert role verification step, and the enhancement step are executed sequentially in a fixed order. Claim 22 A method for quality assurance of a document according to claim 14, wherein the document is a patent specification and the input data includes at least one of a patent opinion, an experimental plan, and experimental data.

Images