An industry large model multi-round dialogue generation method and related equipment
By optimizing the model generation process through task logic parsing and low-rank adaptive techniques, the problem of data scarcity and insufficient evaluation in multi-turn dialogues of large language models in various industries is solved, enabling efficient and reliable multi-turn dialogue applications applicable to fields such as finance, healthcare, law, and government.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SOUTH CHINA UNIV OF TECH
- Filing Date
- 2026-03-03
- Publication Date
- 2026-07-21
AI Technical Summary
In existing technologies, industry-specific large language models face challenges in multi-turn dialogue applications, including a scarcity of high-quality data, difficulty in balancing model optimization efficiency and effectiveness, and an incomplete performance evaluation system for multi-turn dialogue.
Multi-turn dialogue data is generated by parsing task logic. The model generation process is optimized by combining low-rank adaptive technology and direct preference optimization algorithm. A multi-level evaluation system is constructed to achieve efficient and reliable performance evaluation of multi-turn dialogue.
It enables automated conversion from single-round data to multi-round data, reduces computing costs, improves the consistency of task logic and the correctness of decision-making in multi-round interactions, and provides a comprehensive and objective evaluation system applicable to various vertical scenarios such as finance, healthcare, law, and government affairs.
Smart Images
Figure CN122432273A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of artificial intelligence and natural language processing technology, and in particular to a method and related equipment for generating multi-turn dialogues in large industry models. Background Technology
[0002] With significant progress made by large-scale language models in natural language understanding and generation tasks, their application potential in specialized industry scenarios such as finance, government affairs, healthcare, and law is increasingly evident. However, in real-world industry applications, user needs are often complex and dynamically changing, typically requiring multiple rounds of interaction to gradually clarify, refine, and ultimately complete. For example, in the process of a business consultation, generating a diagnostic report, or constructing a complex query, the user's intent is not static but evolves continuously through interaction with the system. However, directly applying existing general-purpose models, primarily based on single-turn question-and-answer, to these multi-turn dialogue scenarios often results in models that are difficult to stably and reliably meet actual business needs.
[0003] Currently, large-scale industry models face the following main technical challenges in multi-turn dialogue applications: First, there is a severe shortage of high-quality, large-scale multi-turn dialogue training data. Most industry datasets exist in the form of single-turn dialogues or instruction-following, implicitly assuming that user needs can be fully expressed in a single input. This data format deviates significantly from the multi-turn interaction pattern of "gradually supplementing and dynamically correcting needs" in real-world business scenarios. This makes it difficult for models trained on such data to effectively learn contextual dependencies and intent evolution patterns, leading to problems such as logical breaks, semantic drift, and error accumulation in multi-turn dialogues.
[0004] Secondly, model optimization strategies struggle to balance task performance and general capabilities. Improving a model's performance on multi-round tasks in specific industries typically requires targeted fine-tuning. However, traditional full-parameter fine-tuning methods are computationally expensive and prone to overfitting on small-scale industry data, even compromising the model's original general language understanding and generation capabilities. Furthermore, guiding the model to prioritize and optimize key decision points that decisively impact task outcomes during complex multi-round inference processes, rather than merely fitting surface-level language patterns, is crucial for improving the model's task execution accuracy and logical consistency. Existing methods lack efficient and controllable optimization mechanisms in this regard.
[0005] Finally, there is a lack of a systematic evaluation framework for multi-turn dialogue scenarios. Existing large-scale model evaluations mostly focus on the final output accuracy of single-turn tasks, making it difficult to comprehensively and objectively measure the model's real-world performance in multi-turn interactions. An excellent industry-wide multi-turn dialogue model not only needs to provide the correct answer in the final turn, but also needs to maintain semantic coherence and logical consistency throughout the dialogue, and possess dynamic adaptive capabilities such as guiding users to supplement key information and responding to user intent corrections. The lack of an evaluation framework makes it difficult to measure the effectiveness of model improvements with a unified standard, and also hinders effective comparison and iterative optimization between different technical solutions.
[0006] Therefore, there is an urgent need for an innovative technical solution that can systematically address the issues of data construction, model optimization, and evaluation and verification in multi-turn dialogues within the industry, so as to enable the reliable and efficient application of large language models in complex and dynamic real-world industry interaction scenarios. Summary of the Invention
[0007] The main objective of this application is to propose a method, electronic device, storage medium, and program product for generating multi-turn dialogues in industry-wide large-scale models based on task parsing-driven decision preference optimization. This aims to solve the technical problems existing in the prior art, such as the scarcity of high-quality multi-turn dialogue data for industry-wide large-scale language models, the difficulty in balancing model optimization efficiency and effectiveness, and the lack of a sound multi-turn dialogue performance evaluation system.
[0008] To achieve the above objectives, one aspect of this application proposes a method for generating multi-turn dialogues in a large industry model, the method comprising: Data construction steps: Obtain the original single-turn dialogue dataset; use the task logic parsing function to structurally decompose the complete task requirements of each sample in the original single-turn dialogue dataset to generate a sequence of staged sub-requirements with logical dependencies; based on the sequence of staged sub-requirements, use a large language model to generate each turn sequentially to construct a multi-turn dialogue dataset. Model optimization steps: Based on freezing the original weights of the large language model, a low-rank adaptive technique is introduced to transform the fine-tuning process into learning low-rank incremental parameters; and a direct preference optimization algorithm is introduced to formalize the multi-round task generation process into a preference-based sequential decision-making process. By constructing preference sample pairs for key decision elements at each task stage, the low-rank incremental parameters are optimized to guide the model to form generation preferences that conform to the task logic at the key decision level; wherein, the key decision elements are information units that have a decisive impact on task progress and final results; Evaluation and verification steps: The optimized model is evaluated using a multi-level evaluation system that includes preference reinforcement strategy evaluation, general performance evaluation, and industry knowledge multi-turn dialogue evaluation.
[0009] In some embodiments, the data construction step includes: Let the original single-turn dialogue dataset be... ,in Indicates the first The complete task requirement descriptions provided by users in each sample. This indicates the corresponding standard output result. Based on the dataset size, the task logic is parsed using a function. Decompose the complete requirement into a sequence of phased sub-requirements:
[0010] in Indicates the first The sample at the th Sub-requirements at each stage Indicates sample The number of sub-requirements obtained by parsing and partitioning; Generate in rounds according to the order of sub-requirements, the first round... The user input for the wheel is constructed as follows:
[0011] in For the complete mission requirements, For the corresponding standard output, These are the sub-requirements that need to be completed in the current round; The system response is generated by the large language model based on the current input and the history of previous dialogues:
[0012] in This represents a pre-trained large language model. Indicates the preceding The context state of a round-robin dialogue; In the final round of guarantees This transforms the original single-turn dataset into a multi-turn dialogue dataset. ,in The multi-turn dialogue sequence corresponding to each sample.
[0013] In some embodiments, the task logic parsing function Based on the way user needs are expressed and their evolutionary characteristics, the complete task requirements are divided into at least one of the following types: Clearly defined requirements: The complete requirements are directly expressed in the first round of input, and the number of sub-requirements is determined after parsing. ; Fuzzy requirement type: The complete requirement is determined by multiple sub-requirements that are gradually supplemented and refined through multiple rounds of interaction. The number of sub-requirements after parsing is... ; Error-corrected requirement type: The complete requirement is generated by a sequence of sub-requirements, including the correction process, and is represented as follows: ,in This indicates the initially proposed incorrect sub-requirement. This indicates the correct sub-requirement that will be corrected in subsequent rounds.
[0014] In some embodiments, the low-rank adaptive technique specifically refers to: for the weight matrix to be fine-tuned in a large model Introducing two low-rank matrices and As trainable incremental parameters, It is a low-rank dimension; the fine-tuned weights are updated as follows: ,in This is a scaling factor used to control the degree of influence of the low-rank incremental weights on the original weights; it is only updated during training. and ,Keep freeze.
[0015] In some embodiments, the construction of the preference-based sequential decision-making process includes: State definition: Sub-requirements for each task phase The current state includes both historical generation results; Action definition: The action defines the set of key decision elements for a given state within that stage. The generative behavior is defined as an action; Policy representation: Parametric model As a strategy, it characterizes the probability distribution of each key decision element in a given state; Preference modeling: for each key decision element Constructing preference sample pairs ,in This indicates the correct output that satisfies both the semantics of the stage task and the constraints of the true objective. This indicates that there is a deviation or logical error in the output.
[0016] In some embodiments, the optimization objective of the direct preference optimization algorithm is to minimize the loss function:
[0017] in This represents the probability distribution of each key decision element in the output given a state. This is the Sigmoid function; this loss function guides the model to assign a higher probability to positive samples and a lower probability to negative samples under the same conditions.
[0018] In some embodiments, the model optimization steps include a combined training strategy: updating only low-rank incremental parameters during training while keeping the original weights frozen; training in the order of the phased sub-demand sequence, with the model output of the previous phase serving as the input context for the next phase; and in each training phase, constructing preference sample pairs for each key decision element identified within that phase and optimizing them using a loss function to explicitly preserve the dependencies between phases.
[0019] In some embodiments, the preference reinforcement strategy evaluation includes: evaluating samples during the training process. In the Key decision elements of the stage Calculate its reward value Accumulated total stage reward Record the dynamic curve of average reward as the number of training steps to quantify the reinforcement effect of preference-driven strategies on model decision-making behavior.
[0020] In some embodiments, the industry knowledge multi-turn dialogue assessment includes at least one of the following sub-tasks: Multi-turn dialogue task: The evaluation object is a complete multi-turn dialogue sequence, and the model generates responses step by step based on the dialogue history. Based on the final answer Evaluate the model's ability to maintain logical consistency and task objectives in long dialogue scenarios; Single-turn dialogue task: Treat each turn in a multi-turn dialogue as an independent sample, and the model input consists of the current sub-need and the historical context. Generate response It is used to measure the model's ability to understand and execute local contexts and phased sub-tasks; Bug fixation task: Focuses on the set of dialogue turns that contain explicit user fixation commands. The model is based on the corrected input. and dialogue history generation response This is used to verify the model's understanding and error correction capabilities of correction instructions.
[0021] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.
[0022] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described above.
[0023] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer program product, including a computer program that, when executed by a processor, implements the method described above.
[0024] The beneficial technical effects achieved by this application include, but are not limited to: 1) Solving the problem of data scarcity: Through task logic parsing and multi-turn dialogue generation, it realizes the automated and low-cost transformation from existing single-turn data to high-quality multi-turn data. The generated data is rich in real task logic evolution characteristics, which significantly improves data utilization efficiency.
[0025] 2) Balancing efficiency and effectiveness: By adopting low-rank adaptive technology, model adaptation can be achieved by updating only a small number of parameters, which greatly reduces computational costs and storage overhead, while effectively avoiding catastrophic forgetting and maintaining the model's generality.
[0026] 3) Precise behavior guidance: By drilling down the direct preference optimization algorithm to the level of "key decision elements", the model's core decision points in multi-round interactions are precisely strengthened, which fundamentally improves the model's task logic consistency and decision correctness, and effectively suppresses error accumulation and propagation.
[0027] 4) Comprehensive evaluation system: A multi-level evaluation system covering process evaluation, basic capability evaluation and task capability evaluation has been constructed, providing quantifiable and traceable guidance for model optimization and filling the gap in multi-round dialogue evaluation in the industry.
[0028] 5) Strong generalization ability: The technical solution of this invention does not depend on a specific industry field. By replacing the task parsing function and industry dataset, it can be quickly migrated to various vertical scenarios such as finance, medical care, law, and government affairs, and has broad industrial application value. Attached Figure Description
[0029] Figure 1 This is a schematic diagram of the system architecture of an embodiment of this application, showing the composition and relationship of the data construction layer, the model optimization layer, and the evaluation and verification layer.
[0030] Figure 2 This is a flowchart of an embodiment of this application, illustrating the main steps and interaction process of the multi-turn dialogue generation method for industry large models based on task parsing-driven decision preference optimization.
[0031] Figure 3 This is a flowchart of the steps of the industry multi-turn dialogue data automatic construction method provided in the embodiments of this application.
[0032] Figure 4 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation
[0033] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.
[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.
[0035] To address the existing technical problems, this application provides an automatic construction method for industry multi-turn dialogue data based on task parsing. Specifically, addressing the issue that existing industry datasets are mainly single-turn and struggle to depict the evolution of needs in real multi-turn interactions, this method builds upon existing high-quality single-turn dialogue datasets by introducing task logic parsing to structurally decompose complex task needs in the original single-turn samples. This parsing process identifies and decomposes the user's complete task needs into a series of staged sub-tasks with clear logical dependencies, such as explicit needs, ambiguous needs requiring gradual supplementation, and erroneous needs requiring dynamic correction—typical interaction patterns. After parsing, the system generates dialogues round by round along the sub-task sequence. In each round of interaction, the model combines the current stage's sub-task description, the complete original task needs, and the standard answer, and, guided by the previous dialogue history, generates the corresponding response for that stage. Through this process, the original single-turn static dataset is systematically and automatically converted into a high-quality multi-turn dialogue dataset, with the final output completely consistent with the target of the original single-turn data. This method enables low-cost data augmentation for dynamic, context-dependent, multi-turn interaction patterns without requiring large-scale manual re-annotation.
[0036] To address the challenge of efficiently learning multi-round task logic, this application proposes a model optimization method combining low-rank adaptation and preference-driven approaches. To reduce computational costs while avoiding overfitting, low-rank adaptation is introduced. This technique freezes most of the original model's parameters, training only a small number of injected low-rank incremental parameters to achieve efficient model fine-tuning. This significantly reduces the number of training parameters while preserving the model's original general knowledge. To further guide the model to make stable and correct key decisions in multi-round interactions, this invention introduces a behavior reinforcement strategy based on direct preference optimization. This strategy treats the multi-round task generation process as a sequence of key decision points. At each task stage, the system identifies key decision elements that decisively influence task progress and constructs "correct" and "incorrect" preference sample pairs for each element. The goal of model optimization is to encourage the model to learn to prioritize generating correct decision outputs within the same context, rather than incorrect or deviating decisions, through a preference loss function. By combining efficient parameter fine-tuning techniques with preference reinforcement for key decision points, the model can significantly enhance its contextual understanding, task reasoning, and logical consistency in multi-turn dialogues while maintaining basic stability in general capabilities, effectively reducing the propagation and accumulation of errors in multi-turn interactions.
[0037] To comprehensively and objectively evaluate the performance of large-scale industry models in multi-turn dialogue scenarios, this application designs a systematic multi-turn dialogue evaluation system. This system not only evaluates the model's final task accuracy but also comprehensively assesses its core capabilities in multi-turn interactions. Specifically, it comprises three components: Preference Reinforcement Strategy Evaluation: This verifies the effectiveness of the model optimization process itself. By monitoring the "reward" trend of key decisions at each stage during training, it quantifies the reinforcement effect of preference-driven strategies on the model's decision-making behavior and the stability of the training process. General Performance Evaluation: This evaluates the impact of the adopted parameter-efficient fine-tuning strategies on the model's original general capabilities (such as understanding, reasoning, and mathematics), ensuring that the optimization process does not lead to a serious degradation of the model's fundamental capabilities. Industry Knowledge Multi-Turn Dialogue Evaluation: This is the core of the evaluation system, aiming to comprehensively evaluate the model's real-world performance in specific industry multi-turn tasks from different granularities. It is further subdivided into three sub-tasks: a multi-turn dialogue task, which evaluates the correctness of the model's final output after completing a full set of multi-turn interactions, focusing on its long-term context management and overall task goal preservation capabilities; a single-turn dialogue task, which evaluates each turn of the multi-turn dialogue as an independent sample, measuring the model's real-time reasoning and execution capabilities in response to local context and phased sub-tasks; and an error correction task, which specifically evaluates the model's response and error correction capabilities when receiving explicit correction instructions from the user, verifying its adaptability and robustness in the face of dynamic changes in intent. This multi-dimensional evaluation system provides a scientific and reliable benchmark for the research, comparison, and continuous improvement of the multi-turn dialogue capabilities of large-scale industry models.
[0038] The solutions of the embodiments of this application will be described in detail below with reference to the accompanying drawings and specific application examples.
[0039] like Figure 1 As shown, the method in this application employs a three-layer system architecture—data construction layer, model optimization layer, and evaluation and verification layer—to achieve intelligent generation of multi-turn dialogues for large industry models. The following section combines... Figure 2 and Figure 3 The process shown below provides a detailed explanation of the specific implementation steps of this application.
[0040] Step S1, Data Construction Layer - Requirement Analysis Based on Task Logic: The system uses a task logic analysis function to structurally decompose the complex task requirements in the original single-turn dialogue dataset, dividing the complete requirements into several staged sub-requirements, and classifying them into three typical types based on the expression method and evolution characteristics of the requirements: explicit requirement type, vague requirement type, and error requirement correction type.
[0041] In one embodiment, the system uses a task logic parsing function. We perform a structured decomposition of the complex task requirements in the original single-turn dialogue dataset. Let the original single-turn dialogue dataset be... ,in Indicates the first The complete task requirement descriptions provided by users in each sample. This indicates the corresponding standard output result. Considering the dataset size, for most industry tasks, the complete requirement can be composed of several phased sub-requirements, thus introducing a task logic parsing function. :
[0042] in Indicates the first The sample at the th Sub-requirements at each stage Indicates sample The number of sub-requirements obtained from the parsing and partitioning. This function can be manually defined by domain experts based on industry task flows, or it can be automatically inferred through a large language model combined with the task description.
[0043] Based on the expression and evolution characteristics of user needs in multi-turn dialogues, task requirements can be divided into three typical types. The first is explicitly defined requirement tasks. These tasks present a complete and explicit requirement at the initial stage of the multi-turn dialogue, and subsequent rounds do not introduce new requirement information, thus eliminating the need for sub-requirement breakdown. In this case, the complete requirement can be directly represented by the input in the first round:
[0044] The second category is tasks with vague requirements. The complete requirement often cannot be clearly defined in a single round of dialogue, but is instead broken down into multiple phased sub-requirements, which are gradually supplemented and refined during multiple rounds of interaction. In this type of task, the complete requirement is jointly determined by the requirements of each phase, and can be formally represented as:
[0045] The third type is the error-correction task. During the gradual formulation of requirements, some sub-requirements may contain errors or be unreasonable due to misunderstandings or insufficient information, requiring correction in subsequent dialogue rounds. In this case, the complete requirement is generated from the sequence of sub-requirements including the correction process, and can be represented as:
[0046] in This indicates the initially proposed incorrect sub-requirement. This indicates the correct sub-requirement that will be corrected in subsequent rounds.
[0047] Step S2, Data Construction Layer - Multi-turn Dialogue Generation: After completing task parsing, the system generates dialogues round by round along the order of sub-requirements. By constructing user input that includes the current sub-requirement, the complete task requirements, and the standard answer, the system guides the large language model to maintain consistency with the task objectives during the round-by-round generation. Finally, the system outputs a result that is consistent with the standard answer of the original single-turn sample, thus realizing the systematic transformation of single-turn data into multi-turn dialogue data.
[0048] In one embodiment, after task parsing is completed, the multi-turn dialogue construction process generates dialogue round by round along the order of sub-requirements. User input in a round-robin dialogue is defined as follows:
[0049] in For the complete mission requirements, For the corresponding standard output, This refers to the sub-requirements that need to be completed in the current round. Providing complete requirements and standard answer information can guide the large language model to maintain consistency of task objectives in each round of generation, improving the accuracy and logical coherence of intermediate results. The system in the... The responses generated by the round are provided by the large language model:
[0050] in This represents a pre-trained large language model. Indicates the preceding The context state of each round of dialogue. By introducing contextual information round by round, the model can stably generate intermediate results for the corresponding stage in each round. In the final round... The model-generated response should be consistent with the standard answer of the original single-round sample: This ensures that the final result obtained from the multi-turn dialogue reorganization is completely consistent with the original single-turn data at the task objective level. The original single-turn dialogue dataset is systematically converted into a multi-turn dialogue dataset:
[0051]
[0052] in The multi-turn dialogue sequence corresponding to each sample.
[0053] Step S3, Model Optimization Layer - Low-Rank Adaptive Technology: By introducing trainable incremental parameters in the form of low-rank matrices on the basis of freezing the original model weights, the fine-tuning process is transformed into learning two low-rank matrices. While maintaining the stability of the original general knowledge and parameter structure, it can efficiently adapt to new industry multi-turn dialogue tasks without the need for full parameter fine-tuning.
[0054] In one embodiment, low-rank adaptive techniques achieve efficient and controllable enhancement of model capabilities by introducing trainable incremental parameters in the form of low-rank matrices, while freezing the original model weights. Given the weight matrix of a linear layer in a large model... LoRA transforms the fine-tuning process into learning two low-rank matrices. and ,in It is a low-rank dimension. The weight update form during the fine-tuning phase is as follows:
[0055] in This is a scaling factor used to control the degree of influence of the low-rank incremental weights on the original weights. By optimizing only... and The model can efficiently adapt to new industry multi-turn dialogue tasks while maintaining the stability of the original general knowledge and parameter structure, without the need for full parameter fine-tuning.
[0056] Step S4, Model Optimization Layer - Preference-Driven Multi-Round Behavior Reinforcement Strategy: Introducing the direct preference optimization algorithm, the multi-round task generation process is formalized into a preference-based sequential decision-making process. Implicit behavioral feedback modeling is performed by constructing preference sample pairs, and the strategy is directly optimized using a loss function to guide the model to form generation preferences that conform to the task logic at the key decision level.
[0057] In one embodiment, to guide the model to form stable and consistent generative behavior during multi-round tasks, a DPO algorithm is introduced to guide the model to make reasonable choices in each round of tasks based on preference samples. Within this framework, the multi-round task generation process can be formalized as a preference-based sequential decision-making process, whose core elements include state, action, policy, reward signal, and optimization objective.
[0058] State represents the context information of the model at the current task stage. For any number of rounds of samples... First, the task logic is parsed using the function. Divide it into a set of ordered, phased sub-requirements:
[0059] in Indicates the first The sub-requirements for each stage are described. Furthermore, the sub-requirements for this stage, together with the generation results of previous rounds, constitute the state input for the current stage, which is used to characterize the progress of the task and the constraints that have been completed, thereby providing a semantic and structural contextual basis for subsequent decisions.
[0060] Based on this, actions correspond to the phased outputs generated by the model in a given state. Specifically, in each phase requirement... Contains key decision elements These elements directly determine the task's progression path and final result, and are the core information that is the focus of attention during training. Therefore, the model in each stage... Actions can be viewed as occurring in the decision space. The process involves sampling and generating sets of key decision-making elements.
[0061] Accordingly, the policy is derived from the parameterized model. This indicates that it characterizes the probability distribution of the model's output for each key decision element under given state conditions. In other words, the goal of policy learning is to continuously generate outputs that are consistent with the task semantics and logically sound at each stage during multiple rounds of interaction, thereby ensuring the consistency and correctness of the overall task behavior.
[0062] To effectively constrain and guide the policy, a reward mechanism is introduced for behavioral feedback modeling. Unlike the explicit numerical rewards in traditional reinforcement learning, this approach uses implicit characterization through preference sample pairs. Specifically, for each key decision element... Construct preference sample pairs ,in This indicates the correct output that satisfies both the semantics of the stage task and the constraints of the true objective. The output indicating a deviation or logical error can be constructed from the original data or obtained by generating perturbations. By modeling a relative relationship of "better / worse," the preferred sample provides the model with stable and easily constructed behavioral feedback signals, enabling each key decision element to receive targeted reinforcement guidance.
[0063] Within the above modeling framework, the optimization objective directly optimizes the policy using the DPO loss function, which is defined as follows:
[0064] in This represents the log probability of the model with respect to the output sequence. The objective function is the Sigmoid function. Under the same state conditions, this function encourages the model to assign higher probabilities to positive sample outputs and lower probabilities to negative sample outputs. This achieves stable reinforcement of multi-round generation behavior without explicit reward modeling or value function estimation, and continuously optimizes the policy towards directions consistent with task logic and real-world constraints at key decision levels. By reinforcing the actions of key decision elements at each stage, the model can gradually form generation preferences consistent with task logic at each stage of the multi-round task, effectively reducing the accumulation and propagation of erroneous decisions and inconsistent behaviors during multi-round interactions.
[0065] Step S5, Model Optimization Layer - Combined Training Strategy: Combine low-rank adaptation with preference-driven reinforcement strategy. During training, only the injected low-rank parameters are updated while the original weights are frozen. Training is performed in the order of stages and preference samples of the corresponding stages are introduced for optimization. The dependencies between stages are explicitly preserved to ensure that the model learns a multi-round decision-making pattern that conforms to the logic of real task advancement.
[0066] In one embodiment, combining LoRA with a preference-driven reinforcement strategy enables efficient fine-tuning of multi-round tasks. For samples... Through task logic parsing function Divide into phased sub-demands And extract key decision elements .
[0067] During training, only the injected low-rank parameters are updated. and Keep the original weights frozen. Meanwhile, for each key decision element... Constructing preference sample pairs By employing a DPO loss optimization strategy, the model prioritizes the correct decision in each stage. Training proceeds sequentially through stages, with the output of the previous stage serving as the input for the next, ensuring logical consistency across multiple rounds. The inference stage also follows a staged generation process, outputting the results of each stage in turn and ultimately generating the complete task output. Through a combination of low-rank parameter tuning and preference-driven optimization, the model maintains its general capabilities while achieving logical consistency and decision correctness across multiple rounds, and reducing redundant or deviating content generated from the objective.
[0068] Step S6, Evaluation and Verification Layer - Preference Reinforcement Strategy Evaluation: By dividing the stage into sub-tasks and extracting key decision elements, construct preference sample pairs and calculate the reward values of key decision elements in each stage, accumulate the total stage reward, record the dynamic curve of the average reward changing with the number of training steps, and observe the trend of key decision reinforcement during model training.
[0069] In one embodiment, to verify the effectiveness of the model optimization method on the constructed dataset, training is performed using multi-turn dialogue samples that cover the task structure at each stage. Each training sample is first processed through a task logic parsing function. Divide the tasks into phased sub-tasks and extract the set of key decision elements within each phase. ,in Indicates the training sample index. Indicates the stage index. Includes all key decision elements in this phase that affect the correctness of the task. For each key decision element Construct preference sample pairs ,in This indicates the correct output that conforms to the true semantics and the stage objectives. Negative samples represent logical errors or deviations from stage-specific decisions.
[0070] During training, the model calculates the reward for each key decision element based on the DPO objective function when generating output at each stage. Specifically, the key decision elements... In the Rewards at each stage Defined as:
[0071] in Indicates the model in parameters The logarithmic probability of the output sequence. The sigmoid function is used to map the difference in probabilities between positive and negative samples to... The interval represents the reward value for a single key decision element within a phase. Total phase reward. The rewards are obtained by accumulating the rewards from all key decision elements within this phase:
[0072] in Indicates training samples In the stage The set of key decision elements in the model. To observe the trend of key decision reinforcement during model training, the total stage reward is calculated every 50 gradient updates during training, and its average value is recorded. This results in a dynamic curve showing how the average reward changes with the number of training steps.
[0073] Step S7, Evaluation and Validation Layer - General Performance Evaluation: Evaluate the model's performance in general capabilities before and after fine-tuning. Use multiple standard datasets covering natural language understanding, reasoning, question answering, and mathematical abilities. By comparing the performance differences before and after fine-tuning, evaluate the impact of the low-rank fine-tuning strategy on the general capabilities of the large model and verify the robustness of the parameter-efficient fine-tuning strategy.
[0074] In one embodiment, fine-tuning a large model may enhance its capabilities for specific tasks, but it may also impact its original general capabilities. To quantify this potential change, the model's performance in general capabilities is evaluated before and after fine-tuning. The evaluation uses multiple standard datasets selected from a general evaluation framework, covering aspects such as natural language understanding, reasoning, question answering, and mathematical ability. By comparing the performance differences before and after fine-tuning, the impact of the LoRA fine-tuning strategy on the general capabilities of the large model can be assessed, thereby verifying the robustness of the parameter-efficient fine-tuning strategy.
[0075] Step S8, Evaluation and Verification Layer - Industry Knowledge Multi-Turn Dialogue Evaluation: Design three types of sub-tasks: multi-turn dialogue tasks, single-turn dialogue tasks, and error correction tasks. Systematically evaluate the model's performance in industry-specific multi-turn dialogue tasks from three aspects: logical consistency at the overall dialogue level, understanding of local context, and response and execution capabilities to user correction instructions.
[0076] In one embodiment, to evaluate the model's performance in industry-specific multi-turn dialogue tasks, three sub-tasks were designed: multi-turn dialogue tasks, single-turn dialogue tasks, and error correction tasks. The model's performance was systematically evaluated from three aspects: overall dialogue-level logical consistency, understanding of local context, and responsiveness and execution of user correction commands. Let the large model be... There are a total of Datasets for multi-turn dialogue Each group of multi-turn dialogue sequences contains wheel Indicates the first User input, This indicates the corresponding standard output result.
[0077] The first task is a multi-turn dialogue task, which requires the model to complete a full set of multi-turn dialogues in a single evaluation. During each turn, the model progressively generates responses based on the current user input and existing dialogue history, incorporating the generated results into the dialogue state update process. Formally, the... Group dialogue in The dialogue history of the wheel is represented as follows:
[0078] Where the model is in the first The round-robin generated response is:
[0079] Multi-turn dialogue tasks use complete dialogue sequences as evaluation objects, focusing on the model's logical consistency and task objective preservation capabilities in long dialogue scenarios. Its performance evaluation is based solely on the final response of each dialogue group. .
[0080] Single-turn dialogue tasks treat each turn in a multi-turn dialogue as an independent evaluation sample, thus avoiding the cumulative impact of errors in previous turns on subsequent reasoning processes. For the first turn... The first in the group dialogue The model input for each round consists of the user needs at the current stage and the historical context:
[0081] The model generates a response based on this:
[0082] This task evaluates all dialogue turns in the dataset to measure the model's ability to understand local contextual information and its reasoning and execution capabilities for phased sub-tasks.
[0083] Error correction tasks, building upon single-turn dialogue tasks, focus on dialogue turns that include explicit user correction commands. Let the first... The set of rounds involving error correction in group dialogues is as follows:
[0084] The model is based on the corrected input information in these rounds. And generate responses from existing dialogue history:
[0085] The error correction task evaluates dialogue rounds that include correction actions, verifying the model's ability to understand and execute user correction instructions, as well as its stability and robustness in correcting errors during the dialogue process.
[0086] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.
[0087] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0088] Please see Figure 4 , Figure 4 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 401 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 402 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 402 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 402 and is called and executed by the processor 401 using the methods described in the embodiments of this application. Input / output interface 403 is used to implement information input and output; The communication interface 404 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 405 transmits information between various components of the device (e.g., processor 401, memory 402, input / output interface 403, and communication interface 404); The processor 401, memory 402, input / output interface 403 and communication interface 404 are connected to each other within the device via bus 405.
[0089] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.
[0090] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.
[0091] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
[0092] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.
[0093] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented in the embodiments of this program product are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments. The executable computer program code or "code" used to perform the various embodiments can be written in high-level programming languages such as C, C++, Python, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (e.g., Transact-SQL), Perl, or in various other programming languages.
[0094] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.
[0095] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.
[0096] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0097] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.
[0098] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0099] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.
[0100] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0101] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0102] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0103] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0104] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.
Claims
1. A method for generating multi-turn dialogues in a large industry model, characterized in that, The method includes the following steps: Data construction steps: Obtain the original single-turn dialogue dataset; use the task logic parsing function to structurally decompose the complete task requirements of each sample in the original single-turn dialogue dataset to generate a sequence of staged sub-requirements with logical dependencies; based on the sequence of staged sub-requirements, use a large language model to generate each turn sequentially to construct a multi-turn dialogue dataset. Model optimization steps: Based on freezing the original weights of the large language model, a low-rank adaptive technique is introduced to transform the fine-tuning process into learning low-rank incremental parameters; and a direct preference optimization algorithm is introduced to formalize the multi-round task generation process into a preference-based sequential decision-making process. By constructing preference sample pairs for key decision elements at each task stage, the low-rank incremental parameters are optimized to guide the model to form generation preferences that conform to the task logic at the key decision level; wherein, the key decision elements are information units that have a decisive impact on task progress and final results; Evaluation and verification steps: The optimized model is evaluated using a multi-level evaluation system that includes preference reinforcement strategy evaluation, general performance evaluation, and industry knowledge multi-turn dialogue evaluation.
2. The method according to claim 1, characterized in that, The data construction steps include: Let the original single-turn dialogue dataset be... ,in Indicates the first The complete task requirement descriptions provided by users in each sample. This indicates the corresponding standard output result. Based on the dataset size, the task logic is parsed using a function. Decompose the complete requirement into a sequence of phased sub-requirements: in Indicates the first The sample at the th Sub-requirements at each stage Indicates sample The number of sub-requirements obtained by parsing and partitioning; Generate in rounds according to the order of sub-requirements, the first round... The user input for the wheel is constructed as follows: in For the complete mission requirements, For the corresponding standard output, These are the sub-requirements that need to be completed in the current round; The system response is generated by the large language model based on the current input and the history of previous dialogues: in This represents a pre-trained large language model. Indicates the preceding The context state of a round-robin dialogue; In the final round of guarantees This transforms the original single-turn dataset into a multi-turn dialogue dataset. ,in The multi-turn dialogue sequence corresponding to each sample.
3. The method according to claim 2, characterized in that, The task logic parsing function Based on the way user needs are expressed and their evolutionary characteristics, the complete task requirements are divided into at least one of the following types: Clearly defined requirements: The complete requirements are directly expressed in the first round of input, and the number of sub-requirements is determined after parsing. ; Fuzzy requirement type: The complete requirement is determined by multiple sub-requirements that are gradually supplemented and refined through multiple rounds of interaction. The number of sub-requirements after parsing is... ; Error-corrected requirement type: The complete requirement is generated by a sequence of sub-requirements, including the correction process, and is represented as follows: ,in This indicates the initially proposed incorrect sub-requirement. This indicates the correct sub-requirement that will be corrected in subsequent rounds.
4. The method according to claim 1, characterized in that, The low-rank adaptive technique specifically involves: for the weight matrix to be fine-tuned in a large model... Introducing two low-rank matrices and As trainable incremental parameters, It is a low-rank dimension; the fine-tuned weights are updated as follows: ,in This is a scaling factor; only updates are performed during training. and ,Keep freeze.
5. The method according to claim 1, characterized in that, The construction of the preference-based sequential decision-making process includes: State definition: Sub-requirements for each task phase The current state includes both historical generation results; Action definition: The action defines the set of key decision elements for a given state within that stage. The generative behavior is defined as an action; Policy representation: Parametric model As a strategy, it characterizes the probability distribution of each key decision element in a given state; Preference modeling: for each key decision element Constructing preference sample pairs ,in This indicates the correct output that satisfies both the semantics of the stage task and the constraints of the true objective. This indicates that there is a deviation or logical error in the output.
6. The method according to claim 5, characterized in that, The optimization objective of the direct preference optimization algorithm is to minimize the loss function: in This represents the probability distribution of each key decision element in the output given a state. This is the Sigmoid function; this loss function guides the model to assign a higher probability to positive samples and a lower probability to negative samples under the same conditions.
7. The method according to claim 1, characterized in that, The model optimization steps include a combined training strategy: during training, only low-rank incremental parameters are updated while the original weights are frozen; training is performed in the order of the phased sub-demand sequence, with the model output of the previous phase serving as the input context for the next phase; in each training phase, preference sample pairs are constructed for each key decision element identified within that phase, and a loss function is used for optimization to explicitly preserve the dependencies between phases.
8. The method according to claim 1, characterized in that, The evaluation of the preference reinforcement strategy includes: for samples during the training process In the Key decision elements of the stage Calculate its reward value Accumulated total stage reward Record the dynamic curve of average reward as the number of training steps to quantify the reinforcement effect of preference-driven strategies on model decision-making behavior.
9. The method according to claim 1, characterized in that, The industry knowledge multi-turn dialogue assessment includes at least one of the following sub-tasks: Multi-turn dialogue task: The evaluation object is a complete multi-turn dialogue sequence, and the model generates responses step by step based on the dialogue history. Based on the final answer Evaluate the model's ability to maintain logical consistency and task objectives in long dialogue scenarios; Single-turn dialogue task: Treat each turn in a multi-turn dialogue as an independent sample, and the model input consists of the current sub-need and the historical context. Generate response It is used to measure the model's ability to understand and execute local contexts and phased sub-tasks; Bug fixation task: Focuses on the set of dialogue turns that contain explicit user fixation commands. The model is based on the corrected input. and dialogue history generation response This is used to verify the model's understanding and error correction capabilities of correction instructions.
10. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 9.