A Method and System for Generating Patent Dependent Claims Based on Reasoning-Aware Evaluation and Feedback Iteration
By employing a reasoning-based perception evaluation and feedback iteration method, and utilizing a pre-trained evaluation model to perform multi-dimensional evaluation and iterative optimization of dependent claims, the problem of uncontrollable generation quality and insufficient evaluation in the prior art is solved, thus achieving high-quality patent dependent claim generation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- DALIAN UNIV OF TECH
- Filing Date
- 2026-04-27
- Publication Date
- 2026-07-31
AI Technical Summary
Existing large language models suffer from problems such as illusory content, deviation from intent, lack of effective evaluation methods, and uncontrollable generation quality when generating patent dependent claims, especially in terms of logical rigor and legal compliance.
The method of reasoning-aware evaluation and feedback iteration is adopted. The dependent claims are evaluated and fed back in multiple dimensions through a pre-trained reasoning-aware evaluation model. Iterative optimization is carried out in combination with a large language model. The optimal result is selected by using an objective function, which simulates the evaluation process of human experts to improve the quality of generation.
It significantly improves the accuracy, relevance, and consistency of dependent claims, reduces redundant clauses, generates quality significantly superior to existing models, and enhances patent drafting efficiency and usability.
Smart Images
Figure CN122491216A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and natural language processing, and to a technique for automatically generating patent documents using a large language model. Specifically, it relates to a method and system for generating patent dependent claims based on reasoning-aware evaluation and feedback iteration. Background Technology
[0002] With the rapid development of Large Language Models (LLMs), their application in the field of patent processing has attracted increasing attention, covering multiple tasks from patent abstract generation to full patent document drafting. Patent drafting is a complex, cross-disciplinary task that requires the generated text to simultaneously meet technical accuracy and legal rigor.
[0003] As a core component of patent documentation, the claims define the scope of patent protection. Dependent claims (DCs) not only need to be logically subordinate to independent claims (ICs), but also need to further define the scope of protection by adding technical features. However, existing methods for directly generating dependent claims using large language models have significant drawbacks:
[0004] 1. Creating illusions and deviating from the intended purpose: Claims generated by existing models often contain a lot of illusory content, deviating from the actual intent of the invention, and often generating redundant or irrelevant clauses.
[0005] 2. Lack of effective evaluation methods: Traditional text generation evaluation metrics (such as BLEU and ROUGE) mainly focus on n-gram overlap, making it difficult to measure the logical rigor and legal compliance of patent claims. Existing LLM-as-a-judge methods often only provide a single score, lacking specific reasoning analysis and improvement suggestions.
[0006] 3. Uncontrollable generation quality: Current generation methods mostly rely on zero-shot or few-shot prompts, lacking a targeted feedback mechanism to guide the model to optimize the generation results.
[0007] Therefore, there is an urgent need for a patent dependent claim generation method that can deeply understand patent logic, provide multi-dimensional reasoning evaluation, and optimize the quality of the generated claims through feedback iteration. Summary of the Invention
[0008] The present invention aims to solve the problems existing in the prior art and provide a method and system for generating patent dependent claims based on reasoning perception evaluation and feedback iteration.
[0009] To achieve the above objectives, the present invention adopts the following technical solution:
[0010] The first aspect of this invention provides a method for generating patent dependent claims based on reasoning-aware evaluation and feedback iteration, comprising:
[0011] Obtain the target independent claim; using a large language model as a generator, generate an initial set of dependent claims based on the target independent claim using preset instructions, and store them in the candidate set;
[0012] The iterative optimization process includes: evaluating the currently generated set of dependent claims using a pre-trained reasoning-aware evaluation model, and outputting evaluation feedback containing multi-dimensional scores and reasoning analysis; inputting the evaluation feedback and the target independent claim into the generator to guide the generator to generate an optimized set of dependent claims and store it in the candidate set;
[0013] Determine whether the preset iteration termination condition has been met. If not, continue the generation and evaluation operations based on the evaluation feedback from the previous round.
[0014] If the iteration termination condition is met, an objective function is constructed, and the optimal set of dependent claims is selected as the final output by maximizing the objective function.
[0015] Furthermore, the multi-dimensional scoring includes accuracy, relevance, completeness, consistency, and language style.
[0016] Furthermore, the reasoning-aware evaluation model uses a general base model as its foundational network, which is trained and constructed to simulate the evaluation process of human experts. It provides evaluation feedback, including multi-dimensional scoring and reasoning analysis, to the generated dependent claims, guiding the iterative optimization of the general large language model. Furthermore, the training process of the reasoning-aware evaluation model includes:
[0017] Utilize multiple large language models to generate "independent claim-dependent claim" sample pairs and reasoning evaluation thought chains;
[0018] Using a base model, an initial evaluation analysis is generated based on sample pairs, including multi-dimensional scoring and inference analysis evaluation feedback, wherein the base model is selected as a strong instruction follower model;
[0019] Human experts were introduced to annotate some sample pairs, while the remaining sample pairs were annotated using existing large language models. The correlation coefficient between the initial evaluation analysis and the annotations was calculated. The top N% of sample pairs with the highest correlation coefficients were retained as a high-quality human dataset. For the remaining sample pairs, the existing large-parameter inference models were used to rewrite and refine them, which were then used as model distillation datasets.
[0020] The pedestal model was initially fine-tuned using a model distillation dataset, and then further fine-tuned using a high-quality human dataset to train the pedestal model by minimizing the negative log-likelihood loss, resulting in a reasoning-aware evaluation model.
[0021] Furthermore, the objective function is as follows:
[0022] ;
[0023] in, For the target independent claim, m is the preset value for the optimal number of dependent claims. Denotes the dependent claim generated in the i-th iteration. express The number of dependent claims, This represents a large language model as a generator. Represents the candidate set The scoring function is the weighted arithmetic mean of the multi-dimensional scores of each dependent claim set.
[0024] A second aspect of the present invention provides a patent dependent claim generation system based on reasoning-aware evaluation and feedback iteration, comprising:
[0025] The acquisition module is used to acquire the target independent claims;
[0026] The generation module is used to generate a set of dependent claims by using a large language model as a generator and by using preset instructions, based on the input target independent claims or the evaluation feedback output by the evaluation module.
[0027] The evaluation module is used to score the generated dependent claim set in multiple dimensions and generate evaluation feedback for reasoning analysis using a pre-trained reasoning-aware evaluation model.
[0028] The iteration control module is used to determine whether the preset iteration termination condition has been met. If not, it controls the generation module and the evaluation module to perform multiple rounds of interaction, inputting the evaluation feedback output by the previous evaluation module into the generation module to optimize the next generation. If the iteration termination condition has been met, the optimal set of dependent claims is selected as the final output result by maximizing the constructed objective function.
[0029] A third aspect of the present invention provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method for generating patent dependent claims based on reasoning-aware evaluation and feedback iteration.
[0030] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described method for generating patent dependent claims based on reasoning-aware evaluation and feedback iteration.
[0031] Compared with the prior art, the present invention has the following advantages:
[0032] 1. High assessment relevance: The inference-aware assessment model has a significantly higher relevance to human expert judgment than existing indicators (such as BLEU, BERTScore, etc.), and can provide accurate quality feedback.
[0033] 2. Improved generation quality: Through the DraFt iteration mechanism, the generated dependent claims outperform the base model and models with larger parameter counts such as GPT-4 and DeepSeek-R1 in terms of correctness and relevance.
[0034] 3. Effectively Reduce Redundancy: This invention can significantly reduce the number of generated redundant claims. Experiments show that for the Llama series models, the number of generated dependent claims can be optimized from 145 to 8, greatly improving the efficiency and usability of patent drafting. Attached Figure Description
[0035] Figure 1 This is a schematic diagram of the overall process of the method of the present invention. Detailed Implementation
[0036] The technical solution of the present invention will be further described below with reference to specific embodiments and accompanying drawings, but the present invention is not limited to specific embodiments.
[0037] A method for generating patent dependent claims based on reasoning-aware evaluation and feedback iteration, the process of which is as follows: Figure 1 As shown, it includes:
[0038] S1, Obtain the target independent claim Set the maximum number of iterations. ; Using large language models As a generator, it uses preset instructions to generate an initial set of dependent claims based on the target independent claims. and store in the candidate set. In this specific embodiment, the large language model is GPT-4o.
[0039] S2, proceeding to the iterative optimization process, including: using a pre-trained Reasoning Aware Evaluation Model (RACE) to evaluate the currently generated set of dependent claims. The evaluation will be conducted, and the output will include multi-dimensional scoring and reasoning analysis. The multi-dimensional scoring includes accuracy, relevance, completeness, consistency, and language style. The evaluation feedback and the target independent claim are input into the generator to guide it in generating an optimized set of dependent claims. Store and update the candidate set For example, the evaluator's evaluation feedback indicates that "claims 24-30 contain redundant or speculative content," and the generator accordingly removes these clauses in the next round.
[0040] The reasoning perception evaluation model uses a general base model as the basic network to simulate the evaluation process of human experts. It provides evaluation feedback, including multi-dimensional scoring and reasoning analysis, to the generated dependent claims, so as to guide the general large language model to perform iterative optimization.
[0041] The training process of the reasoning-aware evaluation model includes:
[0042] 1) Construct a preference-aligned distillation dataset:
[0043] Data Generation: "Independent claim-dependent claim" sample pairs and corresponding reasoning and evaluation thought chains are generated using various large language models of different sizes and architectures. Specifically, in this embodiment, 2,581 "independent claim-dependent claim" sample pairs and corresponding reasoning and evaluation thought chains are generated based on the IPBench dataset using 13 different large language models (covering parameter counts from 7B to 671B).
[0044] Evaluation Trace Generation: Using a base model, initial evaluation analysis is generated for each sample pair based on preset standards, including multi-dimensional scoring and inference analysis evaluation feedback. The base model is a strong instruction follower model. In this embodiment, the base model is Qwen2.5-7B-Instruct. The preset standards include accuracy, relevance, completeness, consistency, and language style.
[0045] Rejection Sampling and Data Cleaning:
[0046] To reduce the cost of manual annotation, human experts are introduced to annotate some sample pairs, while the remaining sample pairs are annotated using one or more existing large language models with good performance. The annotation content includes multi-dimensional scoring and inference analysis evaluation feedback. The annotated evaluation feedback is used to help the base model learn the content and form of human expert evaluation feedback. For example, the GPT-4o model is used to annotate the remaining sample pairs.
[0047] To ensure annotation quality, the correlation coefficient between the initial evaluation analysis and the annotations is calculated. The correlation coefficient is calculated by taking the average of the Pearson correlation coefficients for each dimension of the scores between the initial evaluation analysis and the annotations.
[0048] Screening strategy: The top N% of sample pairs with the highest correlation coefficients and their evaluation feedback are retained. These sample pairs are considered to be highly consistent with human preferences and constitute a high-quality human dataset (D-Human). Specifically, in this embodiment, only the top 8% of sample pairs with the highest correlation coefficients and their evaluation feedback are retained.
[0049] Data augmentation: For the remaining sample pairs, the dependent claims and their evaluation feedback in the sample pairs are rewritten and polished using existing reasoning models with stronger performance and large parameter counts. Specifically, in this embodiment, the existing reasoning model with large parameter counts, DeepSeek-R1, is selected as the existing reasoning model with strong chain-of-thought capabilities and no less than 70B parameters. The modified sample pairs and their evaluation feedback are used as the model distillation dataset (D-Model).
[0050] 2) Model training:
[0051] Distillation Process: A two-stage training approach is employed to minimize the negative log-likelihood loss, enabling the pedestal model to acquire evaluation capabilities that align with human preferences, thereby generating evaluation text that incorporates specific reasoning processes. The first stage of training utilizes the D-Model to perform initial LoRa fine-tuning on the pedestal model for preliminary alignment. The second stage of training utilizes a high-quality D-Human for secondary LoRa fine-tuning to precisely align with human preferences, resulting in a reasoning-aware evaluation model.
[0052] S3, determine whether the preset maximum number of iterations has been reached. If not achieved, the independent claims will be... The generator continues to perform generation and evaluation operations, using the feedback input generator from the previous round.
[0053] S4, if the maximum number of iterations is reached First, the multi-dimensional scores of each dependent claim set output by the reasoning perception evaluation model are calculated, and the weighted arithmetic mean is taken to obtain the scoring function. ;
[0054] Construct the following objective function:
[0055]
[0056] The optimal set of dependent claims is selected as the final output by maximizing the objective function. This objective function is a scoring function that includes a quantity penalty, considering not only the quality score but also constraining the number of generated claims through the penalty term to prevent the generator from generating too many useless dependent claims.
[0057] Experimental results verification
[0058] 1) Validation of the effectiveness of the reasoning perception evaluation model
[0059] On the IPBench dataset, mainstream evaluation metrics such as BLEU, ROUGE-L, METEOR, BERTScore, MoverScore, SimCSE, and LLMScore were selected and compared with the Reasoning Awareness Evaluation Model (RACE) constructed in this embodiment for verification, as shown in Table 1. The results show that compared with other evaluation metrics, the RACE model in this embodiment performs best in Kendall and Spearman, reaching a significance level of 0.0000. Although the NLD performance is not good, the overall evaluation of the RACE model in this embodiment significantly outperforms existing evaluation metrics in terms of accuracy, relevance, and consistency of text evaluation. It achieves better relevance with human judgment, ensures higher quality feedback, and has the evaluation performance and application value to simulate human experts.
[0060] Table 1 Comparison results of RACE model and existing evaluation indicators
[0061]
[0062] 2) Verification of the effectiveness of the method in this embodiment
[0063] On the IPBench dataset, the method of this embodiment was compared with the base model (Qwen2.5-7B-it), GPT-4o, DeepSeek-R1, Llama-3.1-8B-it, and the Llama series models. Evaluations were conducted on multiple metrics including accuracy, relevance, completeness, consistency, language style, and BLEU. As shown in Table 2, the results show that the method of this embodiment significantly outperforms the base model in all five dimensions, including accuracy, relevance, and completeness. Specifically, the accuracy score of this embodiment is 6.96, which is superior to GPT-4o (Accuracy 6.84) and DeepSeek-R1 (Accuracy 6.90), demonstrating outstanding overall performance. Furthermore, regarding the Llama-3.1-8B-it model, in terms of controlling the number of dependent claims generated, the method of this embodiment significantly reduced the average number of dependent claims generated from 145.4 in the Llama-3.1-8B-it model to 8.9, and improved the quality score from 5.11 to 5.97. Through the above optimizations, the method in this embodiment effectively controls the number of generated content and avoids redundant output, while significantly improving the quality of the generated content and greatly enhancing its practicality.
[0064] Table 2 Comparison results between the method of this embodiment and existing methods
[0065]
[0066] This embodiment also provides a patent dependent claim generation system based on reasoning-aware evaluation and feedback iteration, including:
[0067] The acquisition module is used to acquire the target independent claims;
[0068] The generation module is used to generate a set of dependent claims by using a large language model as a generator and by using preset instructions, based on the input target independent claims or the evaluation feedback output by the evaluation module.
[0069] The evaluation module is used to score the generated dependent claim set in multiple dimensions and generate evaluation feedback for reasoning analysis using a pre-trained reasoning-aware evaluation model.
[0070] The iteration control module is used to determine whether the preset iteration termination condition has been met. If not, it controls the generation module and the evaluation module to perform multiple rounds of interaction, inputting the evaluation feedback output by the previous evaluation module into the generation module to optimize the next generation. If the iteration termination condition has been met, the optimal set of dependent claims is selected as the final output result by maximizing the constructed objective function.
[0071] This embodiment also provides an electronic device, including a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described method for generating patent dependent claims based on reasoning-aware evaluation and feedback iteration.
[0072] This embodiment also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described method for generating patent dependent claims based on reasoning-aware evaluation and feedback iteration.
[0073] The above description represents a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A patent dependency model based on reasoning-aware evaluation and feedback iteration The method for generating claims is characterized in that, include: Obtain the target independent claim; using a large language model as a generator, generate an initial set of dependent claims based on the target independent claim using preset instructions, and store them in the candidate set; The iterative optimization process includes: evaluating the currently generated set of dependent claims using a pre-trained reasoning-aware evaluation model, and outputting evaluation feedback containing multi-dimensional scores and reasoning analysis; inputting the evaluation feedback and the target independent claim into the generator to guide the generator to generate an optimized set of dependent claims and store it in the candidate set; Determine whether the preset iteration termination condition has been met. If not, continue the generation and evaluation operations based on the evaluation feedback from the previous round. If the iteration termination condition is met, an objective function is constructed, and the optimal set of dependent claims is selected as the final output by maximizing the objective function.
2. The method according to claim 1, characterized in that, The multi-dimensional scoring includes accuracy, relevance, completeness, consistency, and language style.
3. The method according to claim 1, characterized in that, The reasoning perception evaluation model described above is constructed by using a general base model as the basic network and training it.
4. The method according to claim 1 or 3, characterized in that, The training process of the reasoning-aware evaluation model includes: Utilize multiple large language models to generate "independent claim-dependent claim" sample pairs and reasoning evaluation thought chains; Using a base model, an initial assessment analysis is generated based on sample pairs, including assessment feedback with multi-dimensional scoring and inference analysis; Human experts were introduced to annotate some sample pairs, while the remaining sample pairs were annotated using existing large language models. The correlation coefficient between the initial evaluation analysis and the annotations was calculated. The top N% of sample pairs with the highest correlation coefficients were retained as a high-quality human dataset. For the remaining sample pairs, the existing large-parameter inference models were used to rewrite and refine them, which were then used as model distillation datasets. The pedestal model was initially fine-tuned using the model distillation dataset, and then fine-tuned a second time using a high-quality human dataset to train the pedestal model with the negative log-likelihood loss, thus obtaining the reasoning-aware evaluation model.
5. The method according to claim 3, characterized in that, The base model mentioned above is a strong instruction follower model.
6. The method according to claim 4, characterized in that, The top 8% of sample pairs with the highest correlation coefficients were retained as high-quality human datasets.
7. The method according to claim 1, characterized in that, The objective function is as follows: ； in, For the target independent claim, m is the preset value for the optimal number of dependent claims. Denotes the dependent claim generated in the i-th iteration. express The number of dependent claims, This represents a large language model as a generator. Represents the candidate set The scoring function is the weighted arithmetic mean of the multi-dimensional scores of each dependent claim set.
8. A patent dependency model based on reasoning-aware evaluation and feedback iteration Claim generation system, used to implement the method of any one of claims 1-7, comprising: The acquisition module is used to acquire the target independent claims; The generation module is used to generate a set of dependent claims by using a large language model as a generator and by using preset instructions, based on the input target independent claims or the evaluation feedback output by the evaluation module. The evaluation module is used to score the generated dependent claim set in multiple dimensions and generate evaluation feedback for reasoning analysis using a pre-trained reasoning-aware evaluation model. The iteration control module is used to determine whether the preset iteration termination condition has been met. If not, it controls the generation module and the evaluation module to perform multiple rounds of interaction, inputting the evaluation feedback output by the previous evaluation module into the generation module to optimize the next generation. If the iteration termination condition has been met, the optimal set of dependent claims is selected as the final output result by maximizing the constructed objective function.
9. An electronic device comprising a processor and a memory, wherein the memory stores a computer program, and the processor executes the computer program to implement the method of any one of claims 1-7.
10. A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method of any one of claims 1-7.