Large language model instruction optimization method based on constraint perception self-learning reasoning

Abstract

The invention provides a large language model instruction optimization method based on constraint perception self-learning reasoning, which can be applied to the technical field of natural language processing and artificial intelligence. The method comprises the steps that in the cold start training stage, a thinking chain example generated by a pre-training language model is utilized to assist a target large language model in recognizing constraints in a training instruction set, and the constraints in the training instruction set comprise hard constraints and soft constraints; performing supervision and fine tuning on the target large language model by utilizing the thinking chain example and the training annotation data so as to enable a thinking chain generation process of the target large language model to meet a hard constraint and a soft constraint; in the self-learning training stage, a constraint satisfaction evaluation mechanism is constructed by using the hard constraint and the soft constraint, and a thinking chain generation strategy of the target large language model is optimized through a reinforcement learning algorithm based on the constraint satisfaction evaluation mechanism.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing and artificial intelligence technology, and in particular to a large language model instruction optimization method based on constraint-aware self-learning reasoning. Background Art

[0002] With the widespread application of large language models (LLMs) in the field of natural language processing, command following tasks have become an important benchmark for testing their reasoning and comprehension capabilities. Most existing technologies rely mainly on methods such as supervised fine-tuning (SFT) and direct preference optimization (DPO) to improve the performance of models in simple command following tasks. SFT is usually trained by pairing instructions with standard answers, so that the model learns to generate appropriate responses from input instructions. However, this method has significant limitations when faced with tasks with complex instructions or multiple constraints. In particular, when dealing with complex tasks containing multiple constraints, the model often cannot effectively balance the conflicts between constraints, and its constraint understanding and reasoning capabilities are seriously insufficient, resulting in the generated answers not meeting user expectations. Summary of the Invention

[0003] In view of the above problems, the present invention provides a large language model instruction optimization method based on constraint-aware self-learning reasoning, which is used to solve at least one of the above technical problems.

[0004] According to a first aspect of the present invention, a large language model instruction optimization method based on constraint-aware self-learning reasoning is provided, comprising:

[0005] During the cold-start training phase, the target large language model is assisted in identifying constraints in the training instruction set using thought chain examples generated by the pre-trained language model. The constraints in the training instruction set include hard constraints and soft constraints.

[0006] Use thought chain examples and training annotated data to supervise and fine-tune the target large language model so that the thought chain generation process of the target large language model meets hard constraints and soft constraints;

[0007] During the self-learning training stage, hard constraints and soft constraints are used to construct a constraint satisfaction evaluation mechanism. Based on the constraint satisfaction evaluation mechanism, the thought chain generation strategy of the target large language model is optimized through the reinforcement learning algorithm.

[0008] According to an embodiment of the present invention, the large language model instruction optimization method based on constraint-aware self-learning reasoning further includes:

[0009] Analyze the training instruction set to obtain the constraint set for training the target large language model;

[0010] Constraints that can be quantified in the constraint set are determined as hard constraints, and constraints that cannot be quantified in the constraint set are determined as soft constraints.

[0011] According to an embodiment of the present invention, the above-mentioned thought chain example generated by the pre-trained language model is used to assist the target large language model in identifying the constraints in the training instruction set, including:

[0012] A pre-trained language model is used to generate thought chain examples with an instruction execution reasoning framework. In the thought chain generation strategy of the target large language model, the thought chain examples are used to assist the target large language model in identifying the constraints in the instruction training set, and guide the target large language model to generate thought chain samples that meet the constraints in the training instruction set.

[0013] According to an embodiment of the present invention, the supervised fine-tuning of the thought chain generation process of the target large language model using thought chain examples and training annotated data to meet hard and soft constraints includes:

[0014] Based on the thought chain example, the target large language model is fine-tuned using the training annotated data, so that the target large language model can generate thought chain samples that meet hard constraints and soft constraints based on the preset thought chain generation template, wherein the preset thought chain generation template includes a demand assessment unit and an operation execution unit. The demand assessment unit is used to analyze the instruction set and classify the constraints in the instruction set into hard constraints and soft constraints based on the analysis results.

[0015] According to an embodiment of the present invention, the supervised fine-tuning of the thought chain generation process of the target large language model using thought chain examples and training annotated data to meet hard and soft constraints further includes:

[0016] During the cold start training phase, the thought chain samples are compared with the targets of the training labeled data, and based on the comparison results, the supervised fine-tuning process of the target large language model is optimized using the first preset loss function.

[0017] According to an embodiment of the present invention, during the self-learning training phase, the constraint satisfaction evaluation mechanism is constructed using hard constraints and soft constraints, and based on the constraint satisfaction evaluation mechanism, the thought chain generation strategy of the target large language model is optimized through a reinforcement learning algorithm, including:

[0018] Using the target large language model to generate a thought chain candidate sample set for each training instruction in the training instruction set, and scoring the constraints of each thought chain candidate sample in the thought chain candidate sample set to obtain a score for each thought chain candidate sample;

[0019] Based on hard constraints and soft constraints, a constraint satisfaction evaluation mechanism is constructed by calculating the hard satisfaction of each thought chain candidate sample and obtaining the soft satisfaction based on style matching and tone matching;

[0020] By utilizing the score of each thought chain candidate sample and the constraint satisfaction evaluation mechanism, the thought chain generation strategy of the target large language model is optimized based on the reward value through the reinforcement learning algorithm.

[0021] According to an embodiment of the present invention, the hard satisfaction includes the matching rate of the hard constraints and the quantified value of the quantifiable indicator, and the soft satisfaction includes the semantic similarity and the style matching.

[0022] According to an embodiment of the present invention, the above-mentioned reward value is constructed based on a standard supervised learning loss function;

[0023] Among them, the reward value is used to guide the target large language model in the reinforcement learning algorithm to select the target thought chain sample with the highest reward value from the thought chain candidate sample set.

[0024] According to an embodiment of the present invention, the large language model includes an automatic summarization model of multiple constraint instructions, and the training annotation data includes summary text training data with label values;

[0025] Among them, in the training process of the automatic summarization model of multi-constraint instructions, the hard constraints include the word count constraint of the summary text, the summary text format constraint, the summary language grammar constraint and the entity constraint in the summary;

[0026] Among them, soft constraints include summary style constraints, summary tone constraints, and user preference adaptation constraints.

[0027] According to an embodiment of the present invention, the large language model includes a customer service automatic question-answering system, and the training annotated data includes question-answering training data with label values;

[0028] Among them, during the training process of the customer service automatic question-answering system, hard constraints include the format constraints of the reply text, the authenticity constraints of the reply text, the sensitive word constraints of the reply text, the timeliness constraints, and the user privacy constraints;

[0029] Among them, in the training process of the customer service automatic question-answering system, soft constraints include contextual logic constraints of the reply text, emotional constraints of the reply text, and information coverage constraints of the reply text.

[0030] The large language model instruction optimization method based on constraint-aware self-learning reasoning provided by the present invention achieves fine-grained constraint classification by dividing constraints into soft constraints and hard constraints, thereby being able to more flexibly handle fuzzy constraints and precise constraints in complex instruction sets, ensuring that the generated answers meet the strict requirements of hard constraints while adapting to the flexibility of soft constraints. At the same time, the large language model instruction optimization method based on constraint-aware self-learning reasoning provided by the present invention introduces a self-learning reasoning stage, enabling the target large language model to automatically optimize the generation of thought chains without human intervention, thereby improving the accuracy of constraint analysis and problem solving, and further enhancing the reasoning ability of the target large language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0032] Figure 1 2. This is an application scenario diagram of a large language model instruction optimization method based on constraint-aware self-learning reasoning according to an embodiment of the present invention;

[0033] Figure 2 is a flowchart of a large language model instruction optimization method based on constraint-aware self-learning reasoning according to an embodiment of the present invention;

[0034] Figure 3 2. It is a data processing architecture diagram of a large language model instruction optimization method based on constraint-aware self-learning reasoning according to an embodiment of the present invention;

[0035] Figure 4 is a schematic diagram of a preset thought chain generation template according to an embodiment of the present invention;

[0036] Figure 5 4 is a block diagram of an electronic device suitable for implementing a large language model instruction optimization method based on constraint-aware self-learning reasoning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concept of the present invention.

[0038] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the presence of the features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0039] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0040] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0041] Existing instruction-following technology solutions for large language models have many technical problems: (1) Limitations of multi-constraint instruction processing: Complex instructions often contain multiple constraints, including hard constraints (such as length limits, format requirements) and soft constraints (such as tone, creativity requirements). Existing methods usually treat all constraints as equally important and lack a fine-grained understanding of the differences between constraints, resulting in the model being unable to flexibly handle the ambiguity of soft constraints and the strictness of hard constraints when generating answers. For example, when processing multiple formatting requirements, the model may ignore tone or creativity requirements, causing the generated answers to deviate from the user's actual needs. (2) Insufficient constraint understanding and reasoning: Although existing technical solutions attempt to improve the model's instruction-following capabilities by increasing the diversity of training data or using generative pre-trained language models (such as GPT-4, T5, etc.), these methods still fail to effectively resolve the conflict between complex constraints and fuzzy language. When multiple constraints are intertwined and difficult to quantify, existing models find it difficult to make effective reasoning judgments and thus fail to generate answers that meet all constraints. In addition, most existing methods fail to systematically introduce the Chain-of-Thought (CoT) to explicitly analyze and handle these constraints, lacking a deep understanding of instructions and the ability to reason step by step.

[0042] In order to solve the problems of the existing technology, especially the technical problems faced by the processing of complex instructions with multiple constraints, the present invention innovatively provides a large language model instruction optimization method based on constraint-aware self-learning reasoning by introducing a constraint grading mechanism and self-learning reasoning technology. By distinguishing between hard constraints and soft constraints and introducing reinforcement learning optimization strategies, the ability of large-scale language models in executing multi-constraint instruction tasks is significantly improved, especially in mixed scenarios of processing fuzzy and precise constraints. Through a two-stage training strategy, this method can automatically optimize the model's reasoning ability and constraint processing capabilities, and ultimately generate answers that meet all constraint requirements. Compared with the existing technology, the method provided by the present invention significantly improves the accuracy and robustness of large language models in complex instruction scenarios.

[0043] Figure 1 This is an application scenario diagram of a large language model instruction optimization method based on constraint-aware self-learning reasoning according to an embodiment of the present invention.

[0044] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a natural language processing scenario. A network 104 is used as a medium for providing a communication link between a first terminal device 101, a second terminal device 102, a third terminal device 103, and a server 105. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0045] A user may use a first terminal device 101, a second terminal device 102, or a third terminal device 103 to interact with a server 105 via a network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, or the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (for example only).

[0046] The first terminal device 101 , the second terminal device 102 , and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0047] The server 105 may be a server that provides various services, such as a background management server (for example only) that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process received data such as user requests, and feed back processing results (e.g., web pages, information, or data obtained or generated based on user requests) to the terminal devices.

[0048] It should be noted that the large language model instruction optimization method based on constraint-aware self-learning reasoning provided in the embodiment of the present invention can generally be executed by the server 105. The large language model instruction optimization method based on constraint-aware self-learning reasoning provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0049] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0050] The following will be based on Figure 1 The scene described by Figures 2 to 4 The large language model instruction optimization method based on constraint-aware self-learning reasoning of the disclosed embodiment is described in detail.

[0051] Figure 2 4 is a flowchart of a large language model instruction optimization method based on constraint-aware self-learning reasoning according to an embodiment of the present invention.

[0052] like Figure 2 As shown, the above-mentioned large language model instruction optimization method based on constraint-aware self-learning reasoning includes operations S210 to S230.

[0053] In operation S210, during a cold start training phase, a thought chain example generated by a pre-trained language model is used to assist a target large language model in identifying constraints in a training instruction set, wherein the constraints in the training instruction set include hard constraints and soft constraints.

[0054] The thought chain examples generated by the pre-trained model help the target model identify hard constraints (conditions that must be met, such as the correctness of mathematical formulas) and soft constraints (recommended conditions, such as language fluency) in instructions.

[0055] In operation S220 , supervised fine-tuning is performed on the target large language model using thought chain examples and training annotated data so that the thought chain generation process of the target large language model satisfies hard constraints and soft constraints.

[0056] The model is fine-tuned using labeled data (human-corrected thought chains) to force its output to satisfy two types of constraints.

[0057] In operation S230, during the self-learning training phase, a constraint satisfaction evaluation mechanism is constructed using hard constraints and soft constraints, and based on the constraint satisfaction evaluation mechanism, the thought chain generation strategy of the target large language model is optimized through a reinforcement learning algorithm.

[0058] Build an evaluation mechanism (e.g., number of hard constraint violations, soft constraint scores) and dynamically adjust the generation strategy through reinforcement learning.

[0059] The large language model instruction optimization method based on constraint-aware self-learning reasoning provided by the present invention achieves fine-grained constraint classification by dividing constraints into soft constraints and hard constraints, thereby being able to more flexibly handle fuzzy constraints and precise constraints in complex instruction sets, ensuring that the generated answers meet the strict requirements of hard constraints while adapting to the flexibility of soft constraints. At the same time, the large language model instruction optimization method based on constraint-aware self-learning reasoning provided by the present invention introduces a self-learning reasoning stage, enabling the target large language model to automatically optimize the generation of thought chains without human intervention, thereby improving the accuracy of constraint analysis and problem solving, and further enhancing the reasoning ability of the target large language model.

[0060] Figure 3 This is a data processing architecture diagram of a large language model instruction optimization method based on constraint-aware self-learning reasoning according to an embodiment of the present invention.

[0061] The Constraint-AwaRE Self-Taught Reasoner (CARE-STaR) large language model instruction optimization method provided by the present invention is designed to address the limitations of large-scale language models (LLMs) in processing complex instructions with multiple constraints, especially when processing both soft and hard constraints. Through a two-stage training method, the present invention combines chained reasoning (CoT) and reinforcement learning (RL) techniques, aiming to enable the model to give high-quality answers in complex multi-constraint environments through fine-grained constraint processing and self-optimization. Figure 3 As shown, the above method provided by the present invention includes constraint analysis and classification, and a two-stage training method, which includes a cold start training stage and a self-learning training stage. The current LLM (i.e., the target large language model) uses multiple sampled thought chain examples to generate thought chain samples. Each thought chain includes clear "start", "thought chain content" and "end" identifiers. Each thought chain will be combined with the instruction during the training process and output through the end side. Two strategies are used: using thought chain answering or not using thought chain answering, and different processing methods are adopted for the above two strategies to obtain the final answer text; in the two training stages, two losses are calculated, Loss talk and Loss think Among them, Loss talk Evaluate the overall loss of thought chain generation, and Loss think Focus on the rationality of the thought process. In particular, Loss talk_cot As Loss talkPart of the process that specifically evaluates the degree to which the thought chain satisfies the constraints.

[0062] According to an embodiment of the present invention, the above-mentioned large language model instruction optimization method based on constraint-aware self-learning reasoning also includes: analyzing the training instruction set to obtain a constraint set for training the target large language model; determining the constraints that can be quantified in the constraint set as hard constraints, and determining the constraints that cannot be quantified in the constraint set as soft constraints.

[0063] In the process of constraint analysis and classification, due to the heavy workload of traditional instruction following, all constraints are often treated as equal, without distinguishing different types of constraints. Therefore, this paper proposes a method to distinguish hard constraints from soft constraints.

[0064] Hard constraints: These constraints must be strictly met and usually have clear, quantitative standards, such as word count limits and format requirements. For hard constraints, there are clear criteria for determining whether they are met or not.

[0065] Soft constraints: These constraints impose certain flexibility requirements on the output, usually involving aspects such as style, creative expression, and tone. For example, they require the output to have a formal tone or innovative expression, and allow for a certain range of variation.

[0066] Soft constraints are generally difficult to quantify. The present invention provides the following examples involving soft constraints to help those skilled in the art better understand them. It should be noted that the soft constraints listed in the present invention are exemplary only, and those skilled in the art can determine the hard and soft constraints in the instruction set based on actual needs.

[0067] 1. Language style: such as formality, colloquialism, poetry, humor, restraint, legal terminology, etc.

[0068] 2. Control of tone and intonation: including whether it is friendly, neutral, critical, humble, authoritative, empathetic, etc.

[0069] 3. Diversity and creativity: The model should avoid repetitive expressions, provide novel ideas, or think from multiple perspectives.

[0070] 4. Understanding and responding to ambiguous semantics: Reasonable guesses and smooth handling of ambiguous questions, such as "Do you think XX is good?" This is a vague and subjective question.

[0071] 5. Cultural adaptability and contextual awareness: Make appropriate adjustments based on the region, context, and background, such as differences in Chinese and Western cultural terminology, and holiday sensitivity.

[0072] 6. Aesthetic judgment: This includes the beauty of the copy and the "beauty" of the image description, which is highly subjective.

[0073] 7. Emotional color control: For example, if you want the answer to contain "encouragement", "gentle criticism", "positive and constructive feedback", etc.

[0074] 8. Coherence and sense of rhythm: including writing skills such as natural introduction, development, transition and conclusion, balanced rhythm, and varied sentence structures.

[0075] 9. Value alignment: Although some judgments can be made using hard rules (such as no discrimination), gray-scale judgments on value positions still rely heavily on human labeling or preference feedback.

[0076] 10. Control of knowledge depth or explanation granularity: For example, "concise and clear without losing professionalism" is a subjective balance that is difficult to measure with hard indicators.

[0077] 11. Persuasiveness / persuasion: Whether the copy or suggestion is convincing is also an indicator that relies on human feelings.

[0078] 12. Contextual memory strategy: which information should be retained and which should be downplayed or ignored, relying on fuzzy judgment.

[0079] 13. “Fuzzy appropriate” output length: The requirement is “not too short and not too long”. This type of description itself is a soft constraint.

[0080] 14. Social etiquette and pragmatic appropriateness: such as "whether it appears polite in daily contexts" and "whether it is appropriate in business emails".

[0081] The present invention processes these constraints in a refined manner, adopts strict constraint checking standards for hard constraints, and allows flexible adjustment within a certain range for soft constraints.

[0082] According to an embodiment of the present invention, the above-mentioned use of thought chain examples generated by a pre-trained language model to assist the target large language model in identifying constraints in the training instruction set includes: using the pre-trained language model to generate a thought chain example with an instruction execution reasoning framework, and in the thought chain generation strategy of the target large language model, using the thought chain example to assist the target large language model in identifying constraints in the instruction training set, and guiding the target large language model to generate thought chain samples that meet the constraints in the training instruction set.

[0083] The main purpose of the cold-start training phase is to help the model generate chains of thought (CoT) and learn how to handle constraints in instructions through supervised fine-tuning (SFT). This phase uses pre-trained models (such as GPT-4) to generate preliminary CoT examples and fine-tune with labeled data to gradually improve the model's understanding of multi-constraint instructions.

[0084] The above embodiments involve the CoT generation process during the cold-start training phase: a series of CoT examples are generated using a pre-trained language model (e.g., GPT-4). These CoT examples provide the model with a reasoning framework for instruction execution and help identify how to satisfy the constraints in the instruction through thought chains. The generated CoT should include detailed reasoning steps for the instruction.

[0085] According to an embodiment of the present invention, the above-mentioned supervised fine-tuning of the thought chain generation process of the target large language model to meet hard constraints and soft constraints using thought chain examples and training annotation data includes: based on the thought chain examples, fine-tuning training of the target large language model using training annotation data is performed, so that the target large language model can generate thought chain samples that meet hard constraints and soft constraints based on a preset thought chain generation template, wherein the preset thought chain generation template includes a demand assessment unit and an operation execution unit, and the demand assessment unit is used to analyze the instruction set and classify the constraints in the instruction set into hard constraints and soft constraints based on the analysis results.

[0086] The above-mentioned embodiments involve the supervised fine-tuning process during the cold-start training phase: based on the generated CoT examples, the model is fine-tuned using labeled data to optimize its CoT generation capabilities. During this process, the model learns how to gradually resolve multiple constraints in instructions through thought chains, while simultaneously improving its understanding and generation capabilities. The goal of training is to enable the model to generate the correct thought chain based on a given instruction, satisfying both hard and soft constraints.

[0087] Figure 4 2 is a schematic diagram of a preset thought chain generation template according to an embodiment of the present invention.

[0088] like Figure 4 As shown, the above-mentioned preset thought chain generation template includes two units: demand assessment and answer. The above-mentioned preset thought chain generation template can realize intelligent scene recognition and structured output. The demand assessment unit is used to identify and list all requirements in the instructions. Among them, each requirement is assigned a score (1-5 points): a score of 5 means that it must be strictly followed; a score of 1 means that it can be followed loosely. Use concise key points without additional explanation. The answer unit only lists the simplest steps required to execute the instructions; does not contain analysis, reasoning or checking steps; each step is no more than one sentence; abstract examples are used when formatting is involved. Figure 4 The thought chain generation template shown is an example, which is used to standardize thought chain generation through standardized decomposition → quantitative constraints → disabling redundant thinking → providing output paradigm, avoiding the risk of semantic ambiguity, and to some extent reducing the hardware burden of the target large language model during training. Figure 4 The technical inspiration provided by the thought chain generation template shown can be adopted by other appropriate thought chain generation templates.

[0089] According to an embodiment of the present invention, the above-mentioned supervised fine-tuning of the thought chain generation process of the target large language model to meet hard constraints and soft constraints using thought chain examples and training annotation data also includes: in the cold start training phase, the thought chain samples are compared with the targets of the training annotation data, and based on the comparison results, the supervised fine-tuning process of the target large language model is optimized using a first preset loss function.

[0090] The above embodiments involve the output and feedback of the target large language model during the cold-start training phase: During the cold-start training phase, the CoT generated by the target large language model is compared with the target answer, and a loss function (such as cross-entropy loss) is used to evaluate the effectiveness of the generated CoT. The target large language model adjusts its CoT generation strategy based on the feedback, gradually improving the generated results.

[0091] According to an embodiment of the present invention, during the self-learning training phase, the constraint satisfaction evaluation mechanism is constructed using hard constraints and soft constraints, and based on the constraint satisfaction evaluation mechanism, the thought chain generation strategy of the target large language model is optimized through a reinforcement learning algorithm, including:

[0092] The target large language model is used to generate a thought chain candidate sample set for each training instruction in the training instruction set, and the constraints of each thought chain candidate sample in the thought chain candidate sample set are scored to obtain the score of each thought chain candidate sample; based on hard constraints and soft constraints, a constraint satisfaction evaluation mechanism is constructed by calculating the hard satisfaction of each thought chain candidate sample and obtaining the soft satisfaction based on style matching and tone matching; using the score of each thought chain candidate sample and the constraint satisfaction evaluation mechanism, the target large language model is used to optimize the thought chain generation strategy based on the reward value through the reinforcement learning algorithm.

[0093] According to an embodiment of the present invention, the hard satisfaction includes the matching rate of the hard constraints and the quantified value of the quantifiable indicator, and the soft satisfaction includes the semantic similarity and the style matching.

[0094] According to an embodiment of the present invention, the above-mentioned reward value is constructed based on the standard supervised learning loss function; wherein, the reward value is used to guide the target large language model in the reinforcement learning algorithm to select the target thought chain sample with the highest reward value from the thought chain candidate sample set.

[0095] The self-learning training stage is further described in detail below through specific implementation methods.

[0096] After the cold start training phase is complete, the model enters the self-learning training phase. The core of this phase is to optimize the generation strategy through reinforcement learning. Through reinforcement learning, the target large language model will automatically adjust the CoT generation strategy, thereby improving the accuracy of command execution through a self-feedback mechanism without relying on manually labeled data. The self-learning training phase includes:

[0097] (1) CoT candidate generation: For each input instruction, the model first generates multiple candidate CoTs. Each candidate CoT represents the reasoning process for a different part of the instruction, and the model analyzes the constraints step by step based on these CoTs.

[0098] (2) CoT Evaluation and Constraint Satisfaction Calculation: This paper introduces a constraint satisfaction-based evaluation mechanism to score the quality of each generated CoT. For hard constraints, the target large language model calculates their satisfaction; for soft constraints, the target large language model evaluates their satisfaction by matching the style and tone.

[0099] (3) Reinforcement learning optimization: Based on the CoT score and constraint satisfaction, the target large language model automatically optimizes the CoT generation strategy through a reinforcement learning algorithm. The target large language model assigns a reward value to each CoT generation process and adjusts its generation probability. As shown in formula (1):

[0100] (1),

[0101] in, is a standard supervised learning loss function, and the goal is to fine-tune by maximizing the probability of generating the correct answer. When CoT generation is introduced, the generated thinking chain is used to guide the model to better understand the constraints of the instructions.

[0102] The above formula represents the reward value , which represents the degree to which generating a CoT improves answer prediction accuracy. Based on this reward, the target large language model optimizes its CoT generation strategy so that each CoT better satisfies the constraints in the instruction. Through the self-learning reasoning phase, the model gradually improves its ability to handle multi-constrained instructions, enabling it to select the most appropriate chain of thought from multiple candidate CoTs and generate a final answer that satisfies both hard and soft constraints.

[0103] (4) Multi-round iterative optimization: In each round of training, the target large language model will optimize the generation strategy based on the generated CoT and reward value, so that the target large language model can gradually improve its ability to generate thought chains that meet complex constraints during iteration.

[0104] The present invention verifies the advantages and effectiveness of the above method provided by the present invention through specific experiments. The specific experimental results show that in terms of fine-grained constraint processing: in the IFeval benchmark test, the P-Level index is improved by 6% on average compared with SFT, and the satisfaction of hard constraints (such as specific format, number of chapters, etc.) reaches 89.18% (Mistral-7B); in terms of multi-model adaptability: it supports models with a parameter scale of 3B to 7B (such as Llama-3, Qwen), and the average performance in the CELLO test is improved by 6.1% compared with SFT.

[0105] According to an embodiment of the present invention, the above-mentioned large language model includes an automatic summarization model with multiple constraint instructions, and the training annotation data includes summary text training data with label values; wherein, during the training process of the automatic summarization model with multiple constraint instructions, the hard constraints include summary text word count constraints, summary text format constraints, summary language grammar constraints and summary entity constraints; wherein, the soft constraints include summary style constraints, summary tone constraints and user preference adaptability constraints.

[0106] In the multi-constrained automatic summarization model, the instructions require the generation of a summary that meets format, length, and content requirements. Hard constraints (such as word count and format requirements) and soft constraints (such as creativity and tone requirements) are effectively handled separately. First, a CoT is generated through a cold-start training phase, and the model is fine-tuned through supervision. Next, in the self-learning reasoning training phase, the model generates multiple candidate CoTs and optimizes their generation strategy through reinforcement learning, ultimately generating a summary that meets all constraints.

[0107] According to an embodiment of the present invention, the above-mentioned large language model includes a customer service automatic question-answering system, and the training annotation data includes question-answering training data with label values; wherein, during the training process of the customer service automatic question-answering system, the hard constraints include format constraints of the reply text, authenticity constraints of the reply text, sensitive word constraints of the reply text, timeliness constraints and user privacy constraints; wherein, during the training process of the customer service automatic question-answering system, the soft constraints include contextual logic constraints of the reply text, emotional constraints of the reply text and information coverage constraints of the reply text.

[0108] The system needs to generate responses that meet the format and tone requirements based on the user's needs. Through the CARE-STaR method of this invention, the model can handle complex constraints and generate responses that meet both hard constraints (such as format and word count) and soft constraints (such as tone).

[0109] Figure 5 4 is a block diagram of an electronic device suitable for implementing a large language model instruction optimization method based on constraint-aware self-learning reasoning according to an embodiment of the present invention.

[0110] like Figure 5 As shown, an electronic device 500 according to an embodiment of the present invention includes a processor 501, which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 502 or programs loaded from a storage unit 508 into a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or related chipsets and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0111] Various programs and data required for the operation of the electronic device 500 are stored in the RAM 503. The processor 501, ROM 502, and RAM 503 are connected to each other via a bus 504. The processor 501 executes the programs in the ROM 502 and / or RAM 503 to perform various operations according to the method flow of the embodiment of the present invention. It should be noted that the programs may also be stored in one or more memories other than the ROM 502 and RAM 503. The processor 501 may also execute the programs stored in the one or more memories to perform various operations according to the method flow of the embodiment of the present invention.

[0112] According to an embodiment of the present invention, electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to bus 504. Electronic device 500 may also include one or more of the following components connected to I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 508 including a hard disk; and a communication section 509 including a network interface card such as a LAN card or modem. Communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to I / O interface 505 as needed. Removable media 511, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 510 as needed, so that computer programs read from the removable media can be installed into storage section 508 as needed.

[0113] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0114] According to an embodiment of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, and may include, for example, but not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present invention, a computer-readable storage medium may include the ROM 502 and / or RAM 503 described above, and / or one or more memories other than ROM 502 and RAM 503.

[0115] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0116] It will be understood by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention may be combined and / or coupled in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or couplings fall within the scope of the present invention.

[0117] The above describes embodiments of the present invention. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A large language model instruction optimization method based on constraint-aware self-learning reasoning, characterized in that: The method comprises: During the cold start training phase, the target large language model is assisted in identifying constraints in the training instruction set by using thought chain examples generated by the pre-trained language model, wherein the constraints in the training instruction set include hard constraints and soft constraints; Performing supervised fine-tuning on the target large language model using the thought chain examples and the training annotated data so that the thought chain generation process of the target large language model satisfies the hard constraints and the soft constraints; During the self-learning training phase, the hard constraints and the soft constraints are used to construct a constraint satisfaction evaluation mechanism, and based on the constraint satisfaction evaluation mechanism, the thought chain generation strategy of the target large language model is optimized through a reinforcement learning algorithm.

2. The method according to claim 1, characterized in that The method further comprises: Analyzing the training instruction set to obtain a constraint set for training the target large language model; The constraints in the constraint set that can be quantified are determined as the hard constraints, and the constraints in the constraint set that cannot be quantified are determined as the soft constraints.

3. The method according to claim 1, characterized in that Using thought chain examples generated by the pre-trained language model to assist the target large language model in identifying the constraints in the training instruction set includes: The pre-trained language model is used to generate a thought chain example with an instruction execution reasoning framework. In the thought chain generation strategy of the target large language model, the thought chain example is used to assist the target large language model in identifying the constraints in the instruction training set, and guide the target large language model to generate a thought chain sample that meets the constraints in the training instruction set.

4. The method according to claim 1, wherein Using the thought chain examples and the training annotated data to perform supervised fine-tuning on the thought chain generation process of the target large language model to satisfy the hard constraints and the soft constraints includes: Based on the thought chain example, the target large language model is fine-tuned and trained using the training annotation data, so that the target large language model can generate thought chain samples that meet the hard constraints and the soft constraints based on a preset thought chain generation template, wherein the preset thought chain generation template includes a demand assessment unit and an operation execution unit, and the demand assessment unit is used to analyze the instruction set and classify the constraints in the instruction set into hard constraints and soft constraints based on the analysis results.

5. The method according to claim 4, characterized in that Also includes: During the cold start training phase, the thought chain sample is compared with the target of the training annotated data, and based on the comparison result, the supervised fine-tuning process of the target large language model is optimized using a first preset loss function.

6. The method according to claim 1, characterized in that In the self-learning training phase, the hard constraints and the soft constraints are used to construct a constraint satisfaction evaluation mechanism, and based on the constraint satisfaction evaluation mechanism, the thought chain generation strategy of the target large language model is optimized through a reinforcement learning algorithm, including: generating a thought chain candidate sample set for each training instruction in the training instruction set using the target large language model, and scoring the constraints of each thought chain candidate sample in the thought chain candidate sample set to obtain a score for each thought chain candidate sample; Based on the hard constraints and the soft constraints, the constraint satisfaction evaluation mechanism is constructed by calculating the hard satisfaction of each of the thought chain candidate samples and obtaining the soft satisfaction based on the style matching degree and the tone matching degree; By utilizing the score of each thought chain candidate sample and the constraint satisfaction evaluation mechanism, the thought chain generation strategy of the target large language model is optimized based on the reward value through a reinforcement learning algorithm.

7. The method according to claim 6, characterized in that The hard satisfaction includes the matching rate of the hard constraints and the quantified value of the quantifiable indicator, and the soft satisfaction includes the semantic similarity and style matching.

8. The method according to claim 6, characterized in that The reward value is constructed based on a standard supervised learning loss function; The reward value is used to guide the target large language model in the reinforcement learning algorithm to select the target thought chain sample with the highest reward value from the thought chain candidate sample set.

9. The method according to any one of claims 1 to 8, characterized in that The large language model includes an automatic summary model of multiple constraint instructions, and the training annotation data includes summary text training data with label values; Wherein, in the training process of the automatic summarization model of the multi-constraint instruction, the hard constraints include the word count constraint of the summary text, the format constraint of the summary text, the grammar constraint of the summary language, and the entity constraint in the summary; The soft constraints include summary style constraints, summary tone constraints, and user preference adaptation constraints.

10. The method according to any one of claims 1 to 8, characterized in that The large language model includes a customer service automatic question-answering system, and the training annotation data includes question-answering training data with label values; In the training process of the customer service automatic question-answering system, the hard constraints include format constraints of the reply text, authenticity constraints of the reply text, sensitive word constraints of the reply text, timeliness constraints, and user privacy constraints; In the training process of the customer service automatic question-answering system, the soft constraints include contextual logic constraints of the reply text, emotional constraints of the reply text, and information coverage constraints of the reply text.

Images