A reinforcement learning standard alignment method for automatically determining and optimizing term definitions

Abstract

This invention discloses a reinforcement learning-based standard alignment method for automatically determining and optimizing terminology definitions. The method first constructs a structured instruction fine-tuning dataset integrating expert instructions and standard terminology, and performs supervised fine-tuning on a basic large language model to obtain a concept extraction model and a multi-dimensional judgment model. Second, the concept extraction model is used to analyze the superordinate concepts and distinguishing features of the target term, and the multi-dimensional judgment model automatically evaluates the definition to be optimized across six dimensions: accuracy, conciseness, appropriateness, substitution principle, circular definition principle, and negation form principle. Finally, a composite reward model is constructed, combining the multi-dimensional evaluation results with the semantic similarity to the standard definition, and the GRPO reinforcement learning algorithm is used to iteratively optimize the definition generation strategy until the semantic similarity between the generated revised definition and the standard definition exceeds a preset threshold. This invention achieves automated, standardized, and standard-aligned terminology definitions, significantly improving the quality and consistency of terminology definitions in standardized documents, and is applicable to intelligent standard-setting scenarios in professional fields such as medicine, law, and finance.

Description

Technical Field

[0001] This invention belongs to the field of standard document detection, specifically relating to the fields of natural language processing and artificial intelligence, and particularly to a reinforcement learning standard alignment method for automatically determining and optimizing terminology definitions, applicable to the automated generation, evaluation and refinement of terminology definitions in professional fields such as medicine, law, and finance. Background Technology

[0002] Standards serve as the technical support for economic activities and social development, and are an important component of a nation's fundamental institutions. Standardization plays a fundamental and guiding role in advancing the modernization of the national governance system and governance capacity. Its core carrier—standardization documents—must be scientifically structured, rationally organized, with standardized elements and precise content expression, ultimately forming clear, accurate, and unambiguous technical texts. Within the composition of standardization documents, the "Terms and Definitions" section is of foundational significance; its quality directly affects the consistency of understanding and the effectiveness of implementation of the entire standard.

[0003] According to international and national standards and specifications, terminology definitions must meet several linguistic and logical requirements:

[0004] -Accuracy: Definitions should accurately and completely reflect the essential characteristics of the concepts referred to by the terms;

[0005] - Appropriateness: The definition should contain necessary and sufficient information, neither too broad nor too narrow;

[0006] - Conciseness: Language should be concise, avoiding redundant or vague expressions;

[0007] - No circular definitions: The term to be defined must not be interpreted using itself or its synonyms;

[0008] - Use negation forms with caution: Negation should only be used when the essence of a concept is negative;

[0009] - Conforms to the substitution principle: the definition in the sentence can directly replace the term without changing the meaning.

[0010] However, the drafting of standardization documents is usually led by domain experts, whose advantage lies in their deep expertise, but they often have limitations in terms of linguistic logic, terminological standardization, and standardization writing rules. Writing terminology definitions involves complex language comprehension and logical reasoning abilities, posing a significant challenge to experts without a linguistic background.

[0011] Currently, standards-setting bodies primarily rely on standards quality control personnel to manually review and revise the "terms and definitions" section. This process is highly dependent on expert experience, resulting in inefficiency, subjectivity, and high labor costs, and it is difficult to achieve scalability and consistency. Especially in cross-industry, multilingual, and rapidly iterating standards systems, manual review is no longer sufficient to meet the demands of high-quality and timely standardization.

[0012] In recent years, the rise of Large Language Models (LLMs) has provided new ideas for text generation and understanding tasks.

[0013] Some studies have attempted to use general large models to assist in term definition generation, but serious flaws have been exposed in practice:

[0014] 1. Knowledge blind spots: The general model lacks a systematic understanding of standardization rules, terminological principles, and the conceptual system of specific fields, making it difficult to accurately grasp the core definition structure of "superior concept - distinguishing feature";

[0015] 2. Uncontrollable output: The generated content often has problems such as being verbose, circular definitions, improper use of negation, and deviation from standard expressions, which cannot meet the rigor requirements of standardized documents;

[0016] 3. Illusions and instability: The model is prone to factual errors or logical contradictions, and its output fluctuates greatly under the same input, lacking reliability and repeatability;

[0017] 4. Lack of evaluation-optimization closed loop: Existing methods are mostly single-generation methods and have not established an automatic evaluation and feedback optimization mechanism based on multi-dimensional quality indicators, which makes it impossible to achieve continuous improvement of defined quality.

[0018] It is particularly noteworthy that while existing technologies have explored the use of supervised fine-tuning (SFT) for terminology definition generation, they generally remain at the level of "generation is completion," without introducing a discriminative model that can structurally and multidimensionally evaluate the quality of definitions. This makes it impossible to quantitatively assess core indicators such as "accuracy," "conciseness," and "appropriateness." Meanwhile, some studies have attempted to apply reinforcement learning (RL) to text generation optimization, but their objective functions mostly focus on fluency or semantic relevance, failing to combine the domain constraints unique to terminology definitions (such as extracting superordinate concepts and avoiding circular definitions) with standardized multidimensional evaluation criteria. Consequently, they cannot drive the model to evolve towards "conforming to standards and specifications."

[0019] Therefore, the current technology system still faces the dilemma of "being able to generate, but difficult to evaluate, and unable to modify." There is an urgent need for an intelligent optimization method for terminology definition that can deeply integrate multi-dimensional quality judgment mechanisms and reinforcement learning feedback optimization frameworks, so as to achieve a paradigm leap from "extensive generation" to "precise optimization," thereby truly supporting the construction of standardized documents with high quality, high consistency, and high automation. Summary of the Invention

[0020] This invention provides a reinforcement learning-based standard alignment method for automatically determining and optimizing terminology definitions, aiming to address the problem of difficulty in automatically evaluating and correcting the quality of terminology definitions in standardized documents. This method achieves high-quality generation and standard alignment of terminology definitions by integrating structured semantic extraction, multi-dimensional quality judgment, and reinforcement learning feedback optimization mechanisms.

[0021] Specifically, the method includes the following steps:

[0022] S1. Construct an instruction fine-tuning dataset, and perform supervised fine-tuning of the basic large language model based on the instruction fine-tuning dataset to obtain the concept extraction model M. concept and multi-dimensional judgment model M judge ;

[0023] S11. Collect instructions labeled by domain experts – response pairs to data D expert and definition data from multiple industry standard terminology databases D term Both are formatted into structured triples (I,X,Y), where I is the instruction text, X is the input context, and Y is the expected output, forming the instruction fine-tuning dataset D. sft ;

[0024] S12, in the dataset D sft The basic large language model LLM0 is then subjected to supervised fine-tuning, with the optimization objective being to minimize the negative log-likelihood loss.

[0025] L sft =-E (I,X,Y)～Dsft [logP θ [Y|I,X)]

[0026] After fine-tuning, two specialized sub-models were obtained: Concept Extraction Model M concept Its output is a structured semantic representation of the term (C). hyper C dist ), where C hyper As a superordinate concept, C dist ={c1,c2,...,ck} is the set of distinguishing features, where ck represents the feature element used to distinguish the defined concept from other concepts; multi-dimensional judgment model M judge Its output is a multi-dimensional quality vector q∈{0,1}m m represents the number of evaluation dimensions, with each dimension corresponding to the standardized requirements of accuracy, conciseness, appropriateness, substitution principle, circular definition principle, and negation form principle.

[0027] S2. Obtain the target term t to be optimized, and use the concept extraction model M. concept Extract the key concepts of the target term and use the multi-dimensional judgment model M. judge Perform a multi-dimensional quality assessment on the current definition d to be optimized, and generate multi-dimensional evaluation results;

[0028] S21. Describe the target term t or its context x t Input to concept extraction model M concept To obtain the structured key concepts (C hyper C dist );

[0029] S22, Define the concept to be evaluated as d, and its superordinate concept as C. hyper and the set of distinguishing features C dist Concatenate into a joint input z = Concat(d, C) hyper C dist Input to the multi-dimensional judgment model M judge Output multi-dimensional evaluation results q = [q acc ,q conc ,q mod ,q cha ,q cir ,q not The values ​​of each component are either 0 or 1, representing whether the model satisfies the following dimensions: accuracy, conciseness, appropriateness, substitution principle, circular definition principle, and negation form principle. The evaluation of dimension i is based on the logical output of the internal classification head hi.

[0030] qi=l(σ(hi(z))≧0.5)

[0031] Where σ(·) is the Sigmoid function and l(·) is the indicator function, which means that the value is 1 if the condition is met and 0 if the condition is not met.

[0032] S3. Construct a reinforcement learning optimization framework, based on multi-dimensional evaluation results q and semantic similarity, and iteratively optimize the defined generation strategy π. θ This continues until the semantic similarity between the generated modified definition and the corresponding standard definition exceeds a preset threshold.

[0033] S31. Construct a reward model R(·), and for any candidate d', calculate its reward value as follows:

[0034]

[0035] Where α, β ≧ 0 and α + β = 1, Sim SBERT (·,·) Calculate cosine similarity based on the Sentence-BERT encoder;

[0036] S32. Generate N candidate definitions {d1′,d2′,…,dN′} for the target term t using the current strategy model, which is initially the fine-tuned large language model πθ, and calculate their reward values ​​{r1,r2,…,rN} respectively.

[0037] S33. Use the group relative strategy to optimize the GRPO algorithm to update the policy parameter θ. The optimization objective is to maximize the relative advantage within the group.

[0038]

[0039] in π is the average reward within the group. ref For a fixed reference strategy, λ is the KL divergence regularization coefficient;

[0040] S34. After the policy update, generate the revised definition d. * If Sim is satisfied SBERT (d * ,d std If ) > τ (where τ ∈ [0.9, 0.98]), then the optimization terminates and d is output. * Otherwise, return to the generation step and continue iterating.

[0041] As can be seen from the above technical solution, this invention first constructs a structured instruction fine-tuning dataset containing expert annotation instructions and multi-industry terminology definitions, and then performs supervised fine-tuning of the basic large language model based on this dataset, training a concept extraction model capable of extracting the superordinate concepts and distinguishing features of terms, and a judgment model capable of multi-dimensionally judging the quality of definitions; secondly, for the target terms to be optimized, the concept extraction model is used to obtain their structured semantic representation, and the initial definition and this semantic information are jointly input into the multi-dimensional judgment model to generate a binary evaluation result covering accuracy, conciseness, appropriateness, substitution principle, circular definition principle, and negation form principle; then, a reward model integrating multi-dimensional quality scores and standard definition semantic similarity is constructed to provide reliable feedback signals for reinforcement learning; next, the Group Relative Policy Optimization (GRPO) algorithm is used, combined with the comprehensive reward of candidate definitions and KL divergence constraints, to iteratively optimize the definition generation strategy and achieve continuous improvement in definition quality; finally, the optimization results are verified by setting a semantic similarity threshold (τ∈[0.9,0.98]) to ensure that the output modified definition not only meets the requirements of multi-dimensional specifications but is also highly consistent with authoritative standards. This method has the following significant advantages in the field of intelligent compilation of standardized documents: (1) Multi-dimensional norm-driven discrimination capability: For the first time, the standardized writing rules (such as accuracy, conciseness, appropriateness, substitution principle, no circular definition, and restrictions on the use of negative forms) are formalized into computable multi-dimensional evaluation indicators, and automated, fine-grained quality assessment is achieved through a dedicated discrimination model, overcoming the cognitive blind spot of general large models in terms of the standardization of terminology definitions; (2) Closed-loop reinforcement learning optimization mechanism: A closed-loop optimization framework of "generation-discrimination-reward-policy update-regeneration" is constructed, and the multi-dimensional discrimination results and the standard definition semantic alignment target are integrated into the reinforcement learning reward function, so that the model can be optimized in a targeted manner to a high-quality definition that meets the standardization requirements, effectively suppressing illusion and instability; (3) Standard alignment and verifiable output: By introducing a semantic similarity threshold mechanism based on Sentence-BERT, it is ensured that the final output modified definition is highly consistent with the authoritative standard definition at the semantic level, realizing the generation of quantifiable and verifiable standardized documents, and significantly improving the authority, consistency and practicality of terminology definitions. Attached Figure Description

[0042] After reading the detailed embodiments of the present invention with reference to the accompanying drawings, the reader will gain a clearer understanding of various aspects of the present invention. Wherein:

[0043] Figure 1 This invention illustrates a flowchart of a reinforcement learning standard alignment method for automatically determining and optimizing term definitions, as proposed in this invention.

[0044] Figure 2This diagram illustrates the framework of a reinforcement learning standard alignment method for automatically determining and optimizing term definitions proposed in this invention.

[0045] Figure 3 This invention presents a flowchart of supervised fine-tuning in a reinforcement learning standard alignment method for automatically determining and optimizing term definitions.

[0046] Figure 4 This paper presents a flowchart of the reinforcement learning optimization iteration process in an automatic determination and optimization terminology definition alignment method proposed in this invention. Detailed Implementation

[0047] The present invention proposes a reinforcement learning standard alignment method for automatically determining and optimizing term definitions, which mainly includes the following steps:

[0048] A dedicated discriminant model is constructed to achieve automatic evaluation of terminological semantic parsing and definition quality: By integrating instruction data annotated by domain experts and multi-industry standard terminology databases, a structured instruction fine-tuning dataset is built, and the basic large language model is subjected to supervised fine-tuning to train a concept extraction model and a multi-dimensional judgment model. The former is used to parse the superordinate concepts and distinguishing features of the target term, while the latter performs a structured and quantifiable quality evaluation of the definition to be optimized based on standardized writing specifications (including six dimensions: accuracy, conciseness, appropriateness, substitution principle, no circular definitions, and restrictions on the use of negation forms), and outputs a binary evaluation vector.

[0049] A reinforcement learning optimization framework is constructed to achieve closed-loop iterative optimization of the definition generation strategy: based on the semantic similarity between multi-dimensional evaluation results and candidate and standard definitions, a composite reward function is designed and a reward model is constructed; multiple candidate definitions are generated using the current policy model, and their comprehensive reward value is calculated; the group relative policy optimization (GRPO) algorithm is adopted to maximize the reward while introducing KL divergence constraints to stabilize training and dynamically update the definition generation strategy, so that the model gradually generates higher quality definitions that are more compliant with the specifications and closer to the standard.

[0050] Perform standard alignment verification and output a highly consistent modified definition: After each policy update, generate a modified definition and calculate its semantic similarity with the authoritative standard definition using Sentence-BERT; if the similarity exceeds the preset threshold (τ∈[0.9,0.98]), the optimization is considered successful and the modified definition is output as the final result; otherwise, continue iterative optimization until the alignment condition is met, ensuring that the output result is standardized, accurate and consistent with the standard.

[0051] The specific steps are as follows:

[0052] S1. Construct an instruction fine-tuning dataset, and perform supervised fine-tuning of the basic large language model based on the instruction fine-tuning dataset to obtain the concept extraction model M. concept and multi-dimensional judgment model M judge ;

[0053] S11. Collect instructions labeled by domain experts – response pairs to data D expert and definition data from multiple industry standard terminology databases D term Both are formatted into structured triples (I,X,Y), where I is the instruction text, X is the input context, and Y is the expected output, forming the instruction fine-tuning dataset D. sft ;

[0054] S12, in the dataset D sft The basic large language model LLM0 is then subjected to supervised fine-tuning, with the optimization objective being to minimize the negative log-likelihood loss.

[0055] L sft =-E (I,X,Y)～Dsft [logP θ [Y|I,X)]

[0056] After fine-tuning, two specialized sub-models were obtained: Concept Extraction Model M concept Its output is a structured semantic representation of the term (C). hyper C dist ), where C hyper As a superordinate concept, C dist ={c1,c2,...,ck} is the set of distinguishing features, where ck represents the feature element used to distinguish the defined concept from other concepts; multi-dimensional judgment model M judge Its output is a multi-dimensional quality vector q∈{0,1} m m represents the number of evaluation dimensions, with each dimension corresponding to the standardized requirements of accuracy, conciseness, appropriateness, substitution principle, circular definition principle, and negation form principle.

[0057] S2. Obtain the target term t to be optimized, and use the concept extraction model M. concept Extract the key concepts of the target term and use the multi-dimensional judgment model M. judge Perform a multi-dimensional quality assessment on the current definition d to be optimized, and generate multi-dimensional evaluation results;

[0058] S21. Describe the target term t or its context x t Input to concept extraction model M concept To obtain the structured key concepts (C hyper C dist );

[0059] S22, Define the concept to be evaluated as d, and its superordinate concept as C. hyper and the set of distinguishing features C dist Concatenate into a joint input z = Concat(d, C) hyper C dist Input to the multi-dimensional judgment model M judge Output multi-dimensional evaluation results q = [q acc ,q conc ,q mod ,q cha ,q cir ,q not The values ​​of each component are either 0 or 1, representing whether the model satisfies the following dimensions: accuracy, conciseness, appropriateness, substitution principle, circular definition principle, and negation form principle. The evaluation of dimension i is based on the logical output of the internal classification head hi.

[0060] qi=l(σ(hi(z))≧0.5)

[0061] Where σ(·) is the Sigmoid function and l(·) is the indicator function, which means that the value is 1 if the condition is met and 0 if the condition is not met.

[0062] S3. Construct a reinforcement learning optimization framework, based on multi-dimensional evaluation results q and semantic similarity, and iteratively optimize the defined generation strategy π. θ This continues until the semantic similarity between the generated modified definition and the corresponding standard definition exceeds a preset threshold.

[0063] S31. Construct a reward model R(·), and for any candidate d', calculate its reward value as follows:

[0064]

[0065] Where α, β ≧ 0 and α + β = 1, Sim SBERT (·,·) Calculate cosine similarity based on the Sentence-BERT encoder;

[0066] S32. Generate N candidate definitions {d1′,d2′,…,dN′} for the target term t using the current strategy model, which is initially the fine-tuned large language model πθ, and calculate their reward values ​​{r1,r2,…,rN} respectively.

[0067] S33. Use the group relative strategy to optimize the GRPO algorithm to update the policy parameter θ. The optimization objective is to maximize the relative advantage within the group.

[0068]

[0069] in π is the average reward within the group. refFor a fixed reference strategy, λ is the KL divergence regularization coefficient;

[0070] S34. After the policy update, generate the revised definition d. * If Sim is satisfied SBERT (d * ,d std If ) > τ (where τ ∈ [0.9, 0.98]), then the optimization terminates and d is output. * Otherwise, return to the generation step and continue iterating.

Claims

1. A reinforcement learning standard alignment method for automatically determining and optimizing term definitions, characterized in that... Includes the following steps: S1. Construct an instruction fine-tuning dataset, and perform supervised fine-tuning of the basic large language model based on the instruction fine-tuning dataset to obtain the concept extraction model M. concept and multi-dimensional judgment model M judge ; S2. Obtain the target term t to be optimized, and use the concept extraction model M. concept Extract the key concepts of the target term, including the superordinate concept C. hyper and the set of distinguishing features C dist ={c1,c2,…,ck}, where ck represents the feature elements used to distinguish the defined concept from other concepts, and is determined through the multi-dimensional judgment model M. judge Perform a multi-dimensional quality assessment on the current definition to be optimized, d, where d represents the optimized definition, and generate a multi-dimensional evaluation result q = [q acc ,q conc ,q mod ,q cha ,q cir ,q not ], where each component takes the value 0 or 1, q acc ,q conc ,q mod ,q cha ,q cir ,q not These respectively indicate whether the requirements are met in the dimensions of accuracy, conciseness, appropriateness, substitution principle, circular definition principle, and negative form principle; S3. Construct a reinforcement learning optimization framework, based on multi-dimensional evaluation results q and semantic similarity, and iteratively optimize the defined generation strategy π. θ Until the generated modified definition d * With respect to the corresponding standard definition d std semantic similarity Sim(d * ,d std )>τ, where τ is a preset threshold.

2. The reinforcement learning standard alignment method for automatically determining and optimizing term definitions according to claim 1, characterized in that, Step S1 specifically includes: S11. Collect instructions labeled by domain experts – response pairs to data D expert and definition data from multiple industry standard terminology databases D term Both are formatted into structured triples (I,X,Y), where I is the instruction text, X is the input context, and Y is the expected output, forming the instruction fine-tuning dataset D. sft S12, in the dataset D sft The basic large language model LLM0 is then subjected to supervised fine-tuning, with the optimization objective being to minimize the negative log-likelihood loss. L sft =-E (I,X,Y)～Dsft [logP θ (Y|I,X)], Among them, E (I,X,Y)～Dsft This is represented as the entire instruction fine-tuning dataset D. sft Take the expected value (i.e., average) of all samples in the dataset, P θ (Y|I,X) represents the probability that a model with current parameter θ will generate an expected value Y given instruction I and input X. After fine-tuning, two specialized sub-models were obtained: Concept Extraction Model M concept Its output is a structured semantic representation of the term (C). hyper C dist ), where C hyper As a superordinate concept, C dist ={c1,c2,...,ck} is the set of distinguishing features; multi-dimensional judgment model M judge Its output is a multi-dimensional quality vector q∈{0,1}m, where m is the number of evaluation dimensions. Each dimension corresponds to the standardization requirements of accuracy, simplicity, appropriateness, substitution principle, circular definition principle, and negation form principle.

3. The reinforcement learning standard alignment method for automatically determining and optimizing term definitions according to claim 1, characterized in that, Step S2 specifically includes: S21. Input the target term t or its contextual description xt into the concept extraction model M. concept To obtain the structured key concepts (C hyper C dist ); S22, Define the concept to be evaluated as d, and its superordinate concept as C. hyper and the set of distinguishing features C dist Concatenate into a joint input z = Concat(d, C) hyper C dist Input to the multi-dimensional judgment model M judge Output a multi-dimensional evaluation result q, where the evaluation of each dimension i is based on the logical output of the model's internal classification head hi: qi=l(σ(hi(z))≧0.5) Where σ(·) is the Sigmoid function and l(·) is the indicator function, which means that the value is 1 if the condition is met and 0 if the condition is not met.

4. The reinforcement learning standard alignment method for automatically determining and optimizing term definitions according to claim 1, characterized in that, Step S3 specifically includes: S31. Construct a reward model R(·), and for any candidate d', calculate its reward value as follows: Where α, β ≧ 0 and α + β = 1, Sim SBERT (·,·) Calculate cosine similarity based on the Sentence-BERT encoder; S32. Generate N candidate definitions {d1′,d2′,…,dN′} for the target term t using the current strategy model, which is initially the fine-tuned large language model πθ, and calculate their reward values ​​{r1,r2,…,rN} respectively. S33. Use the group relative strategy to optimize the GRPO algorithm to update the policy parameter θ. The optimization objective is to maximize the relative advantage within the group. in π is the average reward within the group. ref For a fixed reference strategy, λ is the KL divergence regularization coefficient; S34. After the policy update, generate the revised definition d. * If Sim is satisfied SBERT (d * ,d std If ) > τ (where τ ∈ [0.9, 0.98]), then the optimization terminates and d is output. * Otherwise, return to S32 and continue iterating.

Images