Structured text generation model training method and system based on reinforcement learning

Abstract

The application discloses a structured text generation model training method and system based on reinforcement learning, and the method comprises the following steps: constructing a structured instruction data set; fine-tuning a pre-trained large language model to obtain an initial generation model; constructing a lightweight adversarial destruction model to generate three types of error samples, i.e., numerical tampering, text repetition and structure loss, and forming a preference data set; training a multi-task reward model based on the data set, and initializing the weight of each task according to the negative sample proportion, and adaptively adjusting the weight according to the gradient norm change rate; using a proximal policy optimization algorithm to make the KL divergence coefficient dynamically adjust with the comprehensive reward value in sections to balance exploration and stability; constructing a hierarchical prompt word decoupled from a static regulation layer, a dynamic data layer and a structure guiding layer in an inference stage, and cooperating with an incremental grammar analyzer to screen legal tokens during decoding. The application effectively suppresses model hallucinations, and improves data consistency and format compliance rate.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and natural language processing, and in particular to a method and system for training a structured text generation model based on reinforcement learning. Background Technology

[0002] With the rapid development of artificial intelligence technology, Large Language Models (LLMs) have demonstrated powerful capabilities in natural language generation tasks and are gradually being applied to the automated preparation of professional documents such as project feasibility reports and soil and water conservation plans. However, these professional reports have three key characteristics: strong data constraints, strict formatting requirements, and strong legal citations. Core values ​​such as area and volume must be strictly consistent with field survey data, with zero tolerance for error; the chapter structure must fully comply with national or industry standards; and the names and version numbers of laws and regulations must be exactly the same.

[0003] Existing large language models have significant drawbacks when directly applied to this field. First, general-purpose models generate text based on probability, which is prone to "numerical illusions," generating statistical data that appears reasonable but is actually incorrect, resulting in a failure to meet core data consistency requirements. Second, when generating long reports, models easily forget specific chapter structure constraints, causing format drift issues such as missing chapters, incomplete sections, and inconsistent heading levels. Third, traditional RLHF (Reinforcement Learning Based on Human Feedback) relies on manual sorting and labeling of generated text, resulting in extremely high data construction costs and long cycles, making it difficult to scale up in vertical domains. Finally, existing synchronous inference architectures struggle to support concurrent generation by multiple users, resulting in high first-word latency and a poor user experience.

[0004] While industry attempts have attempted to mitigate these issues through Supervised Fine-Tuning (SFT) or cue engineering, these methods only establish weak mappings and cannot fundamentally eliminate illusions and formatting violations. The key bottleneck hindering the practical application of AI in professional reporting lies in enabling models to proactively avoid numerical errors and structural omissions, while ensuring absolute compliance during the inference phase, without relying on expensive manual annotation. Summary of the Invention

[0005] Therefore, the present invention provides a method and system for training a structured text generation model based on reinforcement learning, in order to solve the aforementioned problems existing in the prior art.

[0006] To achieve the above objectives, on the one hand, the present invention provides a method for training a structured text generation model based on reinforcement learning, comprising:

[0007] Step S1: Construct a structured domain instruction dataset, which includes structured input data and canonical output text;

[0008] Step S2: Fine-tune and train the predefined large language model based on the instruction dataset to obtain the initial generated model;

[0009] Step S3: Construct an adversarial destruction model to generate error samples, and pair the error samples with corresponding correct samples to construct a preference dataset, which contains multiple preset error patterns;

[0010] Step S4: Construct a multi-task reward model based on the preference dataset and train it to obtain a target reward model. The target reward model is constructed with the initial generated model as the backbone network and multi-task output branches are added to calculate the independent penalty scores for different error modes.

[0011] Step S5: Optimize the initial generation model with the target reward model. During the optimization process, adjust the policy constraint strength according to the independent penalty score to obtain the target generation model.

[0012] Step S6: Normalize the structured input data to be generated according to the target generation model, construct hierarchical prompt words containing legal constraints, data fields and structural guidance, and introduce a domain-specific syntax filtering mechanism during the decoding process to drive the target generation model to output a structured professional report.

[0013] Furthermore, the process of step S3 includes:

[0014] The adversarial destruction model is formed by replicating the network structure of the initial generation model and reducing the parameter size. The adversarial destruction model optimizes its generation strategy in a gradient ascent manner, learning to apply perturbations to the correct output of the initial generation model while preserving the overall semantics but altering key constraint information, in order to generate the erroneous samples.

[0015] Furthermore, the preset error modes include numerical tampering mode, text duplication mode, and structural missing mode.

[0016] Furthermore, the process of step S4 includes:

[0017] Using the initial generated model as the backbone network, multiple independent fully connected layer branches are connected in parallel at the output of the backbone network, and each fully connected layer branch corresponds to a discrimination task of the preset error mode;

[0018] An initial weight coefficient is set for each of the discrimination tasks, and the initial weight coefficient is determined based on the sample proportion of the corresponding error mode in the preference dataset;

[0019] Calculate the contrastive loss function for each of the discrimination tasks. The contrastive loss function takes the pairs of incorrect and correct samples in the preference dataset as input and updates the backbone network parameters and the parameters of each fully connected layer branch synchronously through gradient backpropagation.

[0020] The initial weight coefficients are dynamically adjusted during training. The adjustment amount of the weight coefficients is negatively correlated with the rate of change of the gradient norm of the corresponding discrimination task on the validation set, so that the target reward model can adaptively enhance the penalty sensitivity to difficult-to-discriminate error patterns.

[0021] Furthermore, the process of dynamically adjusting the initial weight coefficients during training includes:

[0022] Calculate the gradient norm ratio of each discrimination task in the current training batch to the previous training batch. When the ratio is greater than a preset stability threshold, increase the initial weight coefficient of the corresponding discrimination task by an adaptive step size to obtain the target weight coefficient. The adaptive step size is negatively correlated with the natural logarithm of the ratio.

[0023] The target weight coefficients of each discrimination task are normalized using the L2 norm so that the sum of the squares of all weights is 1, in order to prevent a single error mode from dominating the direction of model optimization.

[0024] During training, the entropy value of the target weight coefficient is monitored. When the entropy value is lower than a preset diversity threshold, the adjustment of the target weight coefficient is frozen and an early stop mechanism is triggered.

[0025] Furthermore, the process of step S5 includes:

[0026] Batch input data is sampled from the preference dataset, text trajectories are generated through the initial generation model, the text trajectories are input into the target reward model to obtain the independent penalty scores corresponding to each error mode, and the independent penalty scores are aggregated through a weighted sum function to obtain a comprehensive reward value. The weights of the weighted sum function are dynamically updated according to the historical discrimination accuracy of each error mode.

[0027] When updating the strategy, the KL divergence between the current strategy and the reference strategy is calculated, and a KL divergence penalty term is introduced. The coefficient of the KL divergence penalty term is adaptively adjusted as the overall reward value changes.

[0028] After each strategy update, the sliding variance of the comprehensive reward value is calculated. When the sliding variance of multiple consecutive update cycles is less than a preset convergence threshold, the optimization is terminated and the target generation model is output.

[0029] Furthermore, the adaptive adjustment of the coefficient of the KL divergence penalty term as the overall reward value changes includes:

[0030] When the overall reward value is higher than the preset performance threshold, the coefficient is reduced to relax the constraints; otherwise, the coefficient is increased to tighten the policy update range.

[0031] Furthermore, the hierarchical prompts are composed of a static legal layer, a dynamic data layer, and a structure guidance layer, which are sequentially concatenated. The static legal layer extracts legal provisions related to the current generation task from a preset legal database through semantic retrieval. The dynamic data layer injects the normalized structured input data. The structure guidance layer includes report chapter outlines and appendix placeholders.

[0032] Furthermore, the grammar filtering mechanism constructs an incremental context-free grammar based on domain specifications. In each decoding step, the candidate output tokens are syntactically validated through a predictive analysis table, and only tokens that conform to the domain specifications are retained to enter the sampling set.

[0033] On the other hand, the present invention also provides a training system for a structured text generation model based on reinforcement learning, comprising:

[0034] The data construction module is used to construct a structured instruction dataset, which includes structured input data and standardized output text.

[0035] A supervised fine-tuning module, connected to the data construction module, is used to fine-tune and train a predefined large language model based on the instruction dataset to obtain an initial generated model.

[0036] An adversarial construction module, connected to the supervised fine-tuning module, is used to construct an adversarial destruction model to generate error samples. The error samples are then paired with corresponding correct samples to construct a preference dataset, which contains multiple preset error patterns.

[0037] The reward modeling module, connected to the adversarial construction module, is used to construct a multi-task reward model based on the preference dataset and train it to obtain a target reward model. The target reward model is composed of the initial generated model as the backbone network and multi-task output branches are added to calculate the independent penalty scores for different error modes respectively.

[0038] The strategy optimization module, connected to the reward modeling module, is used to optimize the target reward model for the initial generation model. During the optimization process, the strategy constraint strength is adjusted according to the independent penalty score to obtain the target generation model.

[0039] The report generation module, connected to the strategy optimization module, is used to normalize the structured input data to be generated according to the target generation model, construct hierarchical prompt words containing regulatory constraints, data fields and structural guidance, and introduce a domain-specific syntax filtering mechanism during the decoding process to drive the target generation model to output a structured professional report.

[0040] Compared with existing technologies, the beneficial effects of this invention are as follows: This invention utilizes an adversarial game mechanism between an adversarial destruction model and a multi-task reward model to create a dynamic balance between error sample generation and penalty score feedback. The weight of the error pattern decays to the power of negative one-third of the sample proportion, achieving adaptive reinforcement of difficult-to-discriminate patterns. The KL divergence coefficient is adjusted in segments according to the overall reward performance; higher generation quality results in looser constraints to encourage exploration, while lower quality results in tighter constraints to ensure stability. Simultaneously, the decoupling design of the static regulatory layer and dynamic data layer in the hierarchical prompt words, combined with the incremental grammar analyzer's syntactic validity pruning of tokens during the decoding stage, transforms the formal constraints of domain norms into computable predictive analysis entries. This enables controllable compression of the generation space during inference, allowing the model to maintain generation fluency while achieving extremely high levels of core data consistency and format compliance. Attached Figure Description

[0041] Figure 1 A flowchart illustrating the training method for a structured text generation model based on reinforcement learning provided in an embodiment of the present invention;

[0042] Figure 2 This is a flowchart illustrating step S4 in the reinforcement learning-based structured text generation model training method provided in this embodiment of the invention.

[0043] Figure 3 This is a schematic diagram of the structure of the structured text generation model training method based on reinforcement learning provided in an embodiment of the present invention. Detailed Implementation

[0044] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0045] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0046] It should be noted that in the description of this invention, the terms "upper", "lower", "left", "right", "inner", "outer", etc., which indicate directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings. This is only for the convenience of description and is not intended to indicate or imply that the device or element must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation of this invention.

[0047] Furthermore, it should be noted that, in the description of this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0048] Please see Figure 1 As shown, this invention provides a method for training a structured text generation model based on reinforcement learning, comprising:

[0049] Step S1: Construct a structured domain instruction dataset, which includes structured input data and canonical output text;

[0050] Specifically, this invention is applied to scenarios requiring zero tolerance for numerical accuracy, format standardization, and precise legal citations in the automated compilation of professional reports, such as ecological environment assessments like forest feasibility reports and soil and water conservation plans for construction projects. By employing adversarial reinforcement learning techniques, it overcomes core bottlenecks in professional applications of traditional large language models, such as data illusion, format drift, and high costs of manual annotation, achieving highly reliable intelligent generation of long texts.

[0051] Specifically, reviewed professional reports in the target field are collected as corpus. A JSON Schema data model is defined according to industry standards, containing field types and nesting relationships such as project name, area, volume, and regulatory clause number. Key parameter values ​​are extracted from each report and populated into the JSON Schema to generate a structured input instance. The full text of the report is used as the standardized output text, and three strong constraints—numerical consistency, legal citation integrity, and standard hierarchical structure of chapter titles—are applied to the output text. Data pairs that fail the validation are removed to form an instruction dataset.

[0052] Step S2: Fine-tune and train the predefined large language model based on the instruction dataset to obtain the initial generated model;

[0053] In this embodiment, the LoRA parameter efficient fine-tuning method is used to train the pre-trained large language model, which is the Qwen3-4B-Instruct base model. The LoRA parameters are configured as follows: rank is set to 64, scaling factor is set to 16, and the target module is set to all linear layers. The training hyperparameters are set as follows: batch size is 16, learning rate is 2e-4, training accuracy is BF16, and training epochs are 5. During the fine-tuning training process, equivalent batch processing is achieved through gradient accumulation, enabling the model to establish a strong mapping relationship from the structured input data to the standardized output text. The training loss converges from 1.44 to 0.28, resulting in the initial generative model.

[0054] Step S3: Construct an adversarial destruction model to generate error samples, and pair the error samples with corresponding correct samples to construct a preference dataset, which contains multiple preset error patterns;

[0055] Specifically, step S3 includes the following processes:

[0056] The adversarial destruction model is formed by replicating the network structure of the initial generation model and reducing the parameter size. The adversarial destruction model optimizes its generation strategy in a gradient ascent manner, learning to apply perturbations to the correct output of the initial generation model while preserving the overall semantics but altering key constraint information, in order to generate the erroneous samples.

[0057] Specifically, the adversarial sabotage model is formed by replicating the Transformer network structure of the initial generation model and reducing its parameter size. This parameter size reduction includes halving the hidden layer dimension, reducing the number of attention heads to one-third of the original number, and retaining only the first N / 2 Transformer layers, where N is the total number of layers in the initial generation model. The adversarial sabotage model uses the same Tokenizer and embedding layer configuration as the initial generation model, but does not share weight parameters.

[0058] The adversarial destruction model optimizes the generation strategy using a gradient ascent approach. Specifically, it freezes the parameters of the initial generation model and performs KL divergence maximization optimization on the output of the adversarial destruction model and the output of the initial generation model under the same input conditions. The objective function is:

[0059]

[0060] in, To optimize the objective function, To generate probabilities for adversarial models, λ represents the generation probability of the initial generative model, and λ is the adversarial strength coefficient, ranging from 0.8 to 1.2.

[0061] The AdamW optimizer was used, with a learning rate set to 5e. -5 The batch size is 8, and the training steps are 1000, so that the adversarial destruction model learns to apply a perturbation to the correct output of the initial generation model while preserving the overall semantics but altering the key constraint information.

[0062] Specifically, the preset error modes include numerical tampering mode, text duplication mode, and structural missing mode.

[0063] Specifically, the numerical tampering mode is as follows: the numerical token in the correct sample is identified, the numerical token is determined by matching the combination of numbers and units through regular expressions, Gaussian noise perturbation is applied to the numerical token, the noise standard deviation is set to 5%-10% of the original value, or discretization perturbation is used to randomly replace the last digit of the value with an adjacent digit to generate the erroneous sample with the numerical error.

[0064] Text repetition pattern: Set the repetition penalty parameter repetition_penalty of the adversarial destruction model to 1.0, or add a fixed bias term to the attention weight of the generated tokens during decoding, with the bias value set to 0.3-0.5, to induce the generation of cyclically repeating text segments. If the length of the repeating segment exceeds 10 tokens, it is determined to be a pattern match.

[0065] Structural missing pattern: The correct sample is randomly truncated, and the truncation position is selected evenly in the range of 40%-70% of the total text length, or the entire content is deleted at the chapter level boundary. The sample that does not contain the subheadings that must be included in the standard chapter outline after the missing part constitutes the erroneous sample with structural missing.

[0066] The erroneous samples are paired with their corresponding correct samples in a 1:1 ratio and stored. Each sample pair is labeled with a corresponding error pattern label to form the preference dataset.

[0067] Step S4: Construct a multi-task reward model based on the preference dataset and train it to obtain a target reward model. The target reward model is constructed with the initial generated model as the backbone network and multi-task output branches are added to calculate the independent penalty scores for different error modes.

[0068] Specifically, such as Figure 2 As shown, the process of step S4 includes:

[0069] Step S41: Using the initial generated model as the backbone network, multiple independent fully connected layer branches are connected in parallel at the output of the backbone network. Each fully connected layer branch corresponds to a discrimination task of the preset error mode.

[0070] Specifically, using the Transformer main network of the initial generated model as the backbone, a multilayer perceptron mapping layer is connected to the output layer of the backbone network to map the hidden state vector dimension of the output layer to a d-dimensional intermediate representation, where d is typically 128 or 256. After the multilayer perceptron mapping layer, K independent fully connected layer branches are connected in parallel, where K equals the number of categories of the preset error mode. Each fully connected layer branch has an input dimension of d and an output dimension of 1, corresponding to a single error mode discrimination task.

[0071] Step S42: Set an initial weight coefficient for each of the discrimination tasks. The initial weight coefficient is determined based on the sample proportion of the corresponding error mode in the preference dataset.

[0072] Specifically, initial weight coefficients are set for each of the discrimination tasks: ,in, These are the initial weighting coefficients. denoted as the percentage of samples representing the k-th error pattern in the preference dataset.

[0073] Step S43: Calculate the contrastive loss function for each of the discrimination tasks. The contrastive loss function takes the pairs of incorrect and correct samples in the preference dataset as input and updates the backbone network parameters and the branch parameters of each fully connected layer synchronously through gradient backpropagation.

[0074] Specifically, for the sample pair (c_j, r_j) of the k-th error pattern, where c_j is the correct sample and r_j is the incorrect sample, c_j and r_j are respectively input into the backbone network, passed through the multilayer perceptron mapping layer, and then input into the k-th fully connected layer branch to obtain scores s(c_j) and s(r_j). The contrastive loss function is:

[0075]

[0076] in, Let σ be the contrastive loss function for the k-th error mode, and σ be the sigmoid function. and These are the scores for correct and incorrect samples, respectively.

[0077] The total loss function for the preference dataset is: ,in, This is the total loss function.

[0078] During training, the backbone network parameters θ_backbone and the parameter set {θ_k} of each fully connected layer branch are updated synchronously via gradient backpropagation. The update rule is as follows:

[0079]

[0080] Where θ_backbone represents the backbone network parameters, and η is the learning rate, with a value ranging from 10e. -5 Up to 5×10e -5 ∇ is the gradient operator, representing the vector of partial derivatives of a function with respect to variables.

[0081] Step S44: During training, the initial weight coefficients are dynamically adjusted. The adjustment amount of the weight coefficients is negatively correlated with the rate of change of the gradient norm of the corresponding discrimination task on the validation set, so that the target reward model can adaptively enhance the penalty sensitivity to difficult-to-discern error patterns.

[0082] Specifically, the process of dynamically adjusting the initial weight coefficients during training includes:

[0083] Calculate the gradient norm ratio of each discrimination task in the current training batch to the previous training batch. When the ratio is greater than a preset stability threshold, increase the initial weight coefficient of the corresponding discrimination task by an adaptive step size to obtain the target weight coefficient. The adaptive step size is negatively correlated with the natural logarithm of the ratio.

[0084] Specifically, for each discrimination task k, after the parameter update is completed in each training batch i, the gradient norm corresponding to that batch is calculated. The gradient norm is defined as the L2 norm of the gradient vector of the fully connected layer branch parameters of the discrimination task, i.e. ,in Let be the j-th gradient component in the i-th batch. The gradient norm reflects the strength of the task's influence on model parameter updates. A larger norm indicates higher training difficulty or more unstable learning signal for the current batch. Calculate the ratio of the gradient norm of the current batch to that of the previous batch. : , (Stability threshold) When the gradient for this task fluctuates drastically, it indicates that the model's ability to distinguish this type of error pattern has not yet converged. Therefore, the weight of this task in the total loss function needs to be increased to strengthen the penalty for the corresponding error pattern. The weight coefficients are updated according to the following rules: The adaptive step size is: ,in, To adjust the rate coefficient and control the magnitude of each adjustment.

[0085] The target weight coefficients of each discrimination task are normalized using the L2 norm so that the sum of the squares of all weights is 1, in order to prevent a single error mode from dominating the direction of model optimization.

[0086] Specifically, after each batch of updates, all weight coefficients are normalized: ,in, The normalized target weight coefficients ensure .

[0087] During training, the entropy value of the target weight coefficient is monitored. When the entropy value is lower than a preset diversity threshold, the adjustment of the target weight coefficient is frozen and an early stop mechanism is triggered.

[0088] Specifically, the entropy value of the target weight coefficient is calculated: , where, when the entropy value Below the preset diversity threshold ( This indicates that the weight distribution is too concentrated, and the model may have lost its ability to distinguish multiple error modes in a balanced way. At this time, the early stopping mechanism is triggered to freeze weight updates and terminate training to prevent the model from overfitting to a few error modes.

[0089] Step S5: Optimize the initial generation model with the target reward model. During the optimization process, adjust the policy constraint strength according to the independent penalty score to obtain the target generation model.

[0090] Specifically, step S5 includes the following process:

[0091] Batch input data is sampled from the preference dataset, text trajectories are generated through the initial generation model, the text trajectories are input into the target reward model to obtain the independent penalty scores corresponding to each error mode, and the independent penalty scores are aggregated through a weighted sum function to obtain a comprehensive reward value. The weights of the weighted sum function are dynamically updated according to the historical discrimination accuracy of each error mode.

[0092] Specifically, batches of data are randomly sampled from the preference dataset, with batch sizes ranging from 16 to 32. Each sample contains structured input data and a corresponding standard output text. The structured input data is input into the initial generation model, and a top-p sampling strategy is used to generate text trajectories. The sampling temperature is set to 0.8, the top-p value is set to 0.95, and the maximum generation length is set to 2048 tokens. The text trajectory is the generated complete report text sequence. The generated text trajectory is paired with the standard output text and input into the target reward model to obtain a K-dimensional independent penalty score vector p = [p1, p2, ..., p_K]. Each component corresponds to a penalty value for an error mode; a higher score indicates a more severe error. The weighted sum function aggregates the results to obtain a comprehensive reward value.

[0093]

[0094] in, For the overall reward value, Let k be the independent penalty score for the k-th type of error pattern. This represents the weight of the k-th error mode. Weight Dynamically updated based on the historical accuracy of each error pattern:

[0095]

[0096] in, This represents the accuracy of classifying error pattern k within the t-th update period. Indicates the weight for the next update cycle. This represents the initial weight, with a value of [value]. .

[0097] When updating the strategy, the KL divergence between the current strategy and the reference strategy is calculated, and a KL divergence penalty term is introduced. The coefficient of the KL divergence penalty term is adaptively adjusted as the overall reward value changes.

[0098] Specifically, calculate the current policy Reference Strategy KL divergence: , where s is the input state (prompt word) and a is the generated action (token), used to measure the difference in the distribution of the two policies by the divergence value, to prevent the policy update from deviating too far from the reference policy and causing the model to crash.

[0099] Specifically, the coefficient of the KL divergence penalty term is adaptively adjusted as the overall reward value changes, including:

[0100] When the overall reward value is higher than the preset performance threshold, the coefficient is reduced to relax the constraints; otherwise, the coefficient is increased to tighten the policy update range.

[0101] The coefficient of the KL divergence penalty term The adjustment is adaptive based on the overall reward value, and the adjustment rules are as follows:

[0102]

[0103] in, The coefficients of the KL divergence penalty term, the fundamental coefficients. Preset performance threshold , The relaxation factor is set to 0.2. The tightening coefficient is set to 0.3. When the overall reward value exceeds the performance threshold... Decrease to 0.02, and increase to 0.1 when below the performance threshold.

[0104] After each strategy update, the sliding variance of the comprehensive reward value is calculated. When the sliding variance of multiple consecutive update cycles is less than a preset convergence threshold, the optimization is terminated and the target generation model is output.

[0105] Specifically, a near-end strategy optimization algorithm is used to calculate...

[0106]

[0107] in, This represents the objective function for optimizing the near-end policy. Indicates the expected operator. It is the importance sampling ratio (the ratio of the generation probability of the new strategy to that of the old strategy). The advantage function (measuring the quality of an action) is ε=0.2, which is the clipping parameter. `clip` represents the operation's constraint on the policy update magnitude, preventing gradient explosion. The latter part is the KL penalty term, ensuring that the policy doesn't deviate too far from the reference policy while still achieving high rewards.

[0108] After each round of strategy update, calculate the sliding variance of the overall reward value. :

[0109]

[0110] Where M is the sliding window size, set to 10. This represents the total reward for the first m-1 periods. This represents the average comprehensive reward within the window. The convergence threshold is reached when the sliding variance for five consecutive update cycles is less than the preset convergence threshold. When the time is right, the optimization is terminated and the target generation model is output.

[0111] Step S6: Normalize the structured input data to be generated according to the target generation model, construct hierarchical prompt words containing legal constraints, data fields and structural guidance, and introduce a domain-specific syntax filtering mechanism during the decoding process to drive the target generation model to output a structured professional report.

[0112] Specifically, the structured input data to be generated undergoes three checks and corrections: field type validation, converting string-type values ​​to floating-point numbers according to JSON Schema and retaining a specified number of decimal places; automatic completion of missing values, filling in deducible fields based on knowledge graph reasoning, such as automatically calculating the missing "other forest area" based on the total area and the area of ​​each component; and uniformization of numerical precision, aligning all area fields to hectares and retaining 4 decimal places, and aligning the volume field to cubic meters and retaining 2 decimal places.

[0113] Specifically, the hierarchical prompts are composed of a static legal layer, a dynamic data layer, and a structure guidance layer, which are sequentially concatenated. The static legal layer extracts legal provisions related to the current task from a preset legal database through semantic retrieval. The dynamic data layer injects the normalized structured input data. The structure guidance layer includes report chapter outlines and appendix placeholders.

[0114] Specifically, the static legal layer uses the BM25 sparse coding retrieval tool, takes the currently generated chapter title as the query, extracts the 3-5 most relevant legal provisions from the preset legal database, sets the retrieval relevance threshold to 0.6, and directly embeds the retrieval results into the front end of the prompt words after sorting them by the level of legal validity.

[0115] The dynamic data layer serializes the normalized JSON instance into key-value pair strings and adds numeric fields as needed. <anchor>Tags, add to the regulatory reference field <citation>Labels are injected into the middle of the cue words; these labels are used to reinforce the model's attention to key constraints.

[0116] The structure guidance layer predefines the report chapter outline according to the "LY / T 2492-2015" standard, and embeds the list of mandatory subheadings of the current chapter (such as "1.1 Project Overview, 1.2 Current Status of Forest Land Use") and the appendix placeholder mark [Table:B.1] in Markdown format at the end of the prompt words, clearly informing the model of the structural boundaries of the content to be generated.

[0117] Specifically, the grammar filtering mechanism constructs an incremental context-free grammar based on domain specifications. In each decoding step, the syntactic validity of candidate output tokens is verified through a predictive analysis table, and only tokens that conform to the domain specifications are retained to enter the sampling set.

[0118] Specifically, an incremental context-free grammar G=(V,Σ,P,S) is constructed based on the domain specification, where the terminal symbol set Σ contains all possible chapter title numbers, regulation names, unit words, and numerical patterns, the non-terminal symbol set V defines syntactic units such as chapters, clauses, and numerical sentences, and the production rule P strictly follows the title hierarchy and regulation citation format in the standard specification. The grammar G is compiled into a predictive parsing table M using LL(1) parsing, where each entry M[non-terminal, terminal] indicates a valid next derivation rule.

[0119] During the decoding process, for each candidate output token, the prediction analysis table M is queried: if a valid entry exists in M ​​for the combination of the top non-terminal symbol of the currently generated prefix and the candidate token, then the token is retained and added to the sampling set; otherwise, its probability is set to zero. The sampling set only retains tokens that conform to the grammatical rules, ensuring that the chapter hierarchy and legal citation format of the generated text are absolutely compliant, with a format compliance rate of over 99.98%.

[0120] Specifically, such as Figure 3 As shown, the present invention also provides a training system for a structured text generation model based on reinforcement learning, comprising:

[0121] Data construction module 10 is used to construct a structured domain instruction dataset, which includes structured input data and standardized output text;

[0122] The supervised fine-tuning module 20 is connected to the data construction module 10 and is used to fine-tune the predefined large language model according to the instruction dataset to obtain an initial generated model.

[0123] The adversarial construction module 30 is connected to the supervised fine-tuning module 20 and is used to construct an adversarial destruction model to generate error samples. The error samples are paired with corresponding correct samples to construct a preference dataset, which contains a variety of preset error patterns.

[0124] The reward modeling module 40 is connected to the adversarial construction module 30 and is used to construct a multi-task reward model based on the preference dataset and train it to obtain a target reward model. The target reward model is composed of the initial generated model as the backbone network and multi-task output branches are added to calculate the independent penalty scores for different error modes respectively.

[0125] The strategy optimization module 50 is connected to the reward modeling module 40 and is used to optimize the target reward model for the initial generation model. During the optimization process, the strategy constraint strength is adjusted according to the independent penalty score to obtain the target generation model.

[0126] The report generation module 60, connected to the strategy optimization module 50, is used to normalize the structured input data to be generated according to the target generation model, construct hierarchical prompt words containing regulatory constraints, data fields and structural guidance, and introduce a domain-specific syntax filtering mechanism during the decoding process to drive the target generation model to output a structured professional report.

[0127] Specifically, the structured text generation model training method based on reinforcement learning provided by this invention can execute the structured text generation model training system based on reinforcement learning in the embodiments of this invention and can achieve the same technical effect, which will not be elaborated here.

[0128] Specifically, this invention utilizes an adversarial game mechanism between an adversarial destruction model and a multi-task reward model to create a dynamic balance between error sample generation and penalty score feedback. The weight of the error pattern decays to the power of negative one-third of the sample proportion, achieving adaptive reinforcement of difficult-to-discriminate patterns. The KL divergence coefficient is adjusted in segments according to the overall reward performance; higher generation quality results in looser constraints to encourage exploration, while lower quality results in tighter constraints to ensure stability. Simultaneously, the decoupling design of the static regulatory layer and dynamic data layer in the hierarchical prompt words, combined with the incremental grammar analyzer's syntactic validity pruning of tokens during the decoding phase, transforms the formal constraints of domain norms into computable predictive analysis entries. This enables controllable compression of the generation space during inference, allowing the model to maintain generation fluency while achieving extremely high levels of core data consistency and format compliance.

[0129] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

[0130] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.< / citation> < / anchor>

Claims

1. A method for training a structured text generation model based on reinforcement learning, characterized in that, include: Step S1: Construct a structured domain instruction dataset, which includes structured input data and canonical output text; Step S2: Fine-tune and train the predefined large language model based on the instruction dataset to obtain the initial generated model; Step S3: Construct an adversarial destruction model to generate error samples, and pair the error samples with corresponding correct samples to construct a preference dataset, which contains multiple preset error patterns; The process of step S3 includes: The adversarial destruction model is formed by replicating the network structure of the initial generation model and reducing the parameter size. The adversarial destruction model optimizes its generation strategy in a gradient ascent manner and learns to apply perturbations to the correct output of the initial generation model while preserving the overall semantics but altering key constraint information in order to generate the erroneous samples. Step S4: Construct a multi-task reward model based on the preference dataset and train it to obtain a target reward model. The target reward model is constructed with the initial generated model as the backbone network and multi-task output branches are added to calculate the independent penalty scores for different error modes. Step S5: Optimize the initial generation model with the target reward model. During the optimization process, adjust the policy constraint strength according to the independent penalty score to obtain the target generation model. The process of step S5 includes: Batch input data is sampled from the preference dataset, text trajectories are generated through the initial generation model, the text trajectories are input into the target reward model to obtain the independent penalty scores corresponding to each error mode, and the independent penalty scores are aggregated through a weighted sum function to obtain a comprehensive reward value. The weights of the weighted sum function are dynamically updated according to the historical discrimination accuracy of each error mode. When updating the strategy, the KL divergence between the current strategy and the reference strategy is calculated, and a KL divergence penalty term is introduced. The coefficient of the KL divergence penalty term is adaptively adjusted as the overall reward value changes. After each strategy update, the sliding variance of the comprehensive reward value is calculated. When the sliding variance of multiple consecutive update cycles is less than a preset convergence threshold, the optimization is terminated and the target generation model is output. Step S6: Normalize the structured input data to be generated according to the target generation model, construct hierarchical prompt words containing legal constraints, data fields and structural guidance, and introduce a domain-specific syntax filtering mechanism during the decoding process to drive the target generation model to output a structured professional report.

2. The method for training a structured text generation model based on reinforcement learning according to claim 1, characterized in that, The preset error modes include numerical tampering mode, text duplication mode, and structural missing mode.

3. The method for training a structured text generation model based on reinforcement learning according to claim 2, characterized in that, The process of step S4 includes: Using the initial generated model as the backbone network, multiple independent fully connected layer branches are connected in parallel at the output of the backbone network, and each fully connected layer branch corresponds to a discrimination task of the preset error mode; An initial weight coefficient is set for each of the discrimination tasks, and the initial weight coefficient is determined based on the sample proportion of the corresponding error mode in the preference dataset; Calculate the contrastive loss function for each of the discrimination tasks. The contrastive loss function takes the pairs of incorrect and correct samples in the preference dataset as input and updates the backbone network parameters and the parameters of each fully connected layer branch synchronously through gradient backpropagation. The initial weight coefficients are dynamically adjusted during training. The adjustment amount of the weight coefficients is negatively correlated with the rate of change of the gradient norm of the corresponding discrimination task on the validation set, so that the target reward model can adaptively enhance the penalty sensitivity to difficult-to-discriminate error patterns.

4. The method for training a structured text generation model based on reinforcement learning according to claim 3, characterized in that, The process of dynamically adjusting the initial weight coefficients during training includes: Calculate the gradient norm ratio of each discrimination task in the current training batch to the previous training batch. When the ratio is greater than a preset stability threshold, increase the initial weight coefficient of the corresponding discrimination task by an adaptive step size to obtain the target weight coefficient. The adaptive step size is negatively correlated with the natural logarithm of the ratio. The target weight coefficients of each discrimination task are normalized using the L2 norm so that the sum of the squares of all weights is 1, in order to prevent a single error mode from dominating the direction of model optimization. During training, the entropy value of the target weight coefficient is monitored. When the entropy value is lower than a preset diversity threshold, the adjustment of the target weight coefficient is frozen and an early stop mechanism is triggered.

5. The method for training a structured text generation model based on reinforcement learning according to claim 4, characterized in that, The coefficient of the KL divergence penalty term is adaptively adjusted as the overall reward value changes, including: When the overall reward value is higher than the preset performance threshold, the coefficient is reduced to relax the constraints; otherwise, the coefficient is increased to tighten the policy update range.

6. The method for training a structured text generation model based on reinforcement learning according to claim 5, characterized in that, The hierarchical prompts are composed of a static legal layer, a dynamic data layer, and a structure guidance layer, which are sequentially concatenated. The static legal layer extracts legal provisions related to the current task from a preset legal database through semantic retrieval. The dynamic data layer injects the normalized structured input data. The structure guidance layer includes report chapter outlines and appendix placeholders.

7. The method for training a structured text generation model based on reinforcement learning according to claim 6, characterized in that, The grammar filtering mechanism is based on an incremental context-free grammar constructed according to the domain specification. In each decoding step, the candidate output tokens are syntactically validated through a predictive analysis table, and only tokens that conform to the domain specification are retained to enter the sampling set.

8. A training system for the structured text generation model training method based on reinforcement learning as described in any one of claims 1-7, characterized in that, include: The data construction module is used to construct a structured instruction dataset, which includes structured input data and standardized output text. A supervised fine-tuning module, connected to the data construction module, is used to fine-tune and train a predefined large language model based on the instruction dataset to obtain an initial generated model. An adversarial construction module, connected to the supervised fine-tuning module, is used to construct an adversarial destruction model to generate error samples. The error samples are then paired with corresponding correct samples to construct a preference dataset, which contains multiple preset error patterns. The reward modeling module, connected to the adversarial construction module, is used to construct a multi-task reward model based on the preference dataset and train it to obtain a target reward model. The target reward model is composed of the initial generated model as the backbone network and multi-task output branches are added to calculate the independent penalty scores for different error modes respectively. The strategy optimization module, connected to the reward modeling module, is used to optimize the target reward model for the initial generation model. During the optimization process, the strategy constraint strength is adjusted according to the independent penalty score to obtain the target generation model. The report generation module, connected to the strategy optimization module, is used to normalize the structured input data to be generated according to the target generation model, construct hierarchical prompt words containing regulatory constraints, data fields and structural guidance, and introduce a domain-specific syntax filtering mechanism during the decoding process to drive the target generation model to output a structured professional report.

Images