Multi-agent based instruction quality evaluation method, medium, device and product

Abstract

This invention discloses a method, medium, device, and product for instruction quality assessment based on multi-agent systems, relating to the field of instruction quality assessment. The method includes: constructing a multi-agent collaborative assessment framework consisting of four types of agents: a planning agent, an evaluation agent, a negotiation agent, and a decision-making agent; the planning agent analyzing the instruction-input-output triplet to generate instruction metadata, and adaptively generating an instruction quality assessment plan based on the instruction metadata; the evaluation agent calculating and outputting the assessment results according to the set of assessment indicators and their corresponding weights in the instruction quality assessment plan; the negotiation agent executing the negotiation strategy specified by the planning agent based on the assessment results and outputting the negotiation results; and the decision-making agent generating a final instruction quality decision score, decision level, and chain-reasoning explanation based on the instruction metadata, the instruction quality assessment plan, the assessment results, and the negotiation results. This invention can solve the problem of static and fixed assessment processes and negotiation strategies.

Description

Technical Field

[0001] This invention relates to the field of instruction quality assessment technology, and in particular to instruction quality assessment methods, media, devices and products based on multi-agent systems. Background Technology

[0002] With the widespread application of large language models in natural language processing, they have demonstrated powerful language understanding and generation capabilities after pre-training. However, in real-world scenarios, single pre-trained models often require fine-tuning with instructions to meet diverse and fine-grained task requirements. Instruction quality evaluation, as a key step in fine-tuning data quality and model performance, must encompass multiple dimensions such as structural regularity, semantic accuracy, and logical rationality, while also considering efficiency, interpretability, and automation. Traditional manual evaluation and statistical or machine learning metrics have limitations in certain dimensions.

[0003] Current research often employs a single-agent framework, centered on the idea of ​​a large language model as the evaluator. This approach utilizes a single large language model as the evaluator, designing prompts to allow the model to score and interpret the quality of instructions. While this method offers high automation and some scalability, providing results similar to human judgment in some open-ended generation tasks, its singular evaluation perspective makes it difficult to meet the diverse needs of instructions in complex scenarios, and it is susceptible to model bias and prompt design issues.

[0004] Further research shows that multi-agent systems can exchange viewpoints and question each other in multiple rounds of discussion, and reach a consensus under the aggregation of the referee agent. This significantly improves the accuracy, robustness, and interpretability of the evaluation, thereby enabling knowledge memorization, enhancing long-term planning, and integrating the expertise of multiple models. It can effectively solve complex multi-step tasks, improve the interaction efficiency between different agents, and achieve collective intelligence.

[0005] However, current multi-agent evaluation methods still have several limitations: First, evaluation metrics mostly adopt static and fixed designs, lacking the ability to dynamically adjust according to tasks and environments; second, the evaluation system fails to coordinate effectively, with general metrics and domain-specific or task-specific metrics often being isolated or simply superimposed, lacking a phased and hierarchical overall design; third, negotiation strategies between agents mostly rely on pre-configured static settings, making it difficult to adaptively adjust according to the actual category, domain, and complexity of the task; furthermore, existing evaluations usually focus on the final results output by the agents, failing to effectively evaluate the clarity, completeness, and executability of the task instructions themselves, resulting in an incomplete evaluation scope; finally, most evaluation processes rely on end-to-end interaction, lacking explicit and structured intermediate representations of the evaluation planning process, leading to opaque evaluation logic, difficulty in process traceability and human intervention, which is detrimental to system debugging and optimization.

[0006] The aforementioned shortcomings make it difficult for existing technologies to build a comprehensive, flexible, interpretable, and intervention-enabled automatic instruction quality evaluation system, thus restricting the large-scale construction and application of high-quality instruction fine-tuning data. Summary of the Invention

[0007] The purpose of this invention is to address the problem of static and fixed evaluation processes and negotiation strategies in existing instruction quality assessment methods, and to propose a multi-agent-based instruction quality assessment method, comprising the following steps:

[0008] Construct a multi-agent collaborative evaluation framework consisting of four types of agents: planning agent, evaluation agent, negotiation agent, and decision-making agent; The planning agent analyzes the instruction-input-output triplet, generates instruction metadata, and adaptively generates instruction quality assessment planning based on the instruction metadata; The evaluation agent calculates and outputs the evaluation results based on the set of evaluation indicators and the weights of each indicator in the instruction quality evaluation plan. Based on the evaluation results, the negotiating agent executes the negotiation strategy specified by the planning agent and outputs the negotiation results; Based on the instruction metadata, instruction quality assessment plan, assessment results, and negotiation results, the decision-making agent generates the final instruction quality decision score, decision level, and chain reasoning explanation.

[0009] Furthermore, the instruction metadata includes: task category, subcategory, application domain, and instruction complexity.

[0010] Furthermore, the instruction quality assessment plan includes: a set of assessment indicators and their weights, negotiation strategy types, and allocation information for corresponding execution agents.

[0011] Furthermore, negotiation strategies include: When the instruction involves text classification, label determination, ambiguous restatement, or preference selection, a voting merging strategy is adopted; the negotiation result under the voting merging strategy is:

[0012] in, This represents the negotiation result under the voting merge strategy, where q represents the instruction-input-output triple. This represents the voting merge strategy function. For the evaluation results, For the first Natural language statements of a negotiating intelligent agent {Support, Neutral, Oppose} indicates the first... Discrete position labels of a negotiating agent; To negotiate the total number of intelligent agents; When the instruction involves rating, ranking, preference strength judgment, or requires probability output, a confidence-weighted strategy is adopted; the negotiation result under the confidence-weighted strategy is:

[0013] in, This indicates the negotiation result under the confidence-weighted strategy. This represents the confidence-weighted strategy function. express The corresponding confidence level; When the instructions involve subjective tasks, logical reasoning, or multi-perspective arguments, a multi-round debate strategy is adopted; the negotiation results under the multi-round debate strategy are as follows:

[0014] in, This indicates the outcome of negotiations following a multi-round debate strategy. This represents the multi-round debate strategy function. Let R represent the natural language description of the m-th deliberative agent in the i-th round, and let R represent the debate round number. When the instruction involves fact-checking, policy and regulatory Q&A, or scientific and academic knowledge, an evidence retrieval strategy is adopted. The negotiation result under the evidence retrieval strategy is as follows:

[0015] in, This indicates the negotiation result under the evidence retrieval strategy. This represents the evidence retrieval strategy function. Indicates the first The set of evidence cited by a negotiating agent.

[0016] Furthermore, the evaluation agent includes a basic evaluation agent and a specific evaluation agent. The basic evaluation agent is configured to perform programmable computation of statistical and model-based evaluation metrics on instruction-input-output triples. The specific evaluation agent is configured to guide the large language model to score metrics based on the large language model through prompting engineering.

[0017] Furthermore, the planning agent is a fine-tuned large language model. First, based on a manually labeled standard planning sample set, low-rank adaptation technology is used for fine-tuning, enabling the large language model to learn and generate dynamic instruction quality assessment plans that conform to a predefined structure; the training sample format is... Where q represents the instruction-input-output triple, These are standard planning samples with manual annotations; Then, a preference dataset is constructed, and preference alignment is performed using the direct preference optimization algorithm.

[0018] Furthermore, the loss for minimizing preference alignment using the direct preference optimization algorithm is defined as:

[0019] in, This represents the loss minimized when using the direct preference optimization algorithm for preference alignment. The basic model parameters representing the frozen state of the large language model. This represents the trainable low-rank adaptation parameters during the direct preference optimization phase. Represents a preference dataset, This represents a sample of preference data in the preference dataset. Standard planning representing artificial structures, This represents the plan generated by the model. Indicates in Expectation under the given conditions The sigmoid function represents the model's decision. Superior Confidence level, This is a temperature coefficient used to adjust the model's sensitivity to differences in preferences. Represents the implicit reward function;

[0020] in, This indicates that the direct preference optimization stage of the large language model selects when the triple is q. The strategy function, These represent the effective parameters in the direct preference optimization phase of the large language model. This indicates that the direct preference optimization stage of the large language model selects when the triple is q. The strategy function, This indicates that during the fine-tuning phase of the large language model, the choice is made when the triple is q. The strategy function, This indicates that during the fine-tuning phase of the large language model, the choice is made when the triple is q. The strategy function, These represent the effective parameters during the fine-tuning phase of the large language model.

[0021] The present invention also proposes a computer-readable storage medium storing a computer program, characterized in that: when the computer program is executed by a processor, it implements the above-described method.

[0022] The present invention also proposes an electronic device, characterized in that it includes a processor and a memory, the processor being interconnected with the memory, wherein the memory is used to store a computer program, the computer program including computer-readable instructions, and the processor is configured to invoke the computer-readable instructions to execute the above-described method.

[0023] The present invention also proposes a computer program product, including a computer program / instruction, characterized in that the computer program / instruction implements the above-described method when executed by a processor.

[0024] The beneficial effects of the technical solution provided by this invention are: (1) This invention solves the problem of traditional single agent evaluation being limited by a single perspective and susceptible to model bias by constructing a multi-agent framework that coordinates four modules: planning, evaluation, negotiation, and decision-making. This significantly improves the comprehensiveness, robustness, and objectivity of the evaluation results.

[0025] (2) The planning agent of the present invention automatically generates a structured evaluation plan based on the task category, application field and instruction complexity by dynamically parsing the input instruction-input-output triplet, so that the evaluation scheme accurately matches the task characteristics, avoids the defect of planning and task disconnection in traditional methods, and solves the problem of rigid evaluation task planning and inability to dynamically adapt to task characteristics.

[0026] (3) This invention uses a task-feature-driven dynamic negotiation strategy selection mechanism to intelligently match four strategies: vote merging, confidence weighting, multi-round debate, and evidence retrieval. At the same time, it mandates that each stage outputs a structured record containing reasoning basis, evidence chain, and intermediate results, forming a fully traceable and auditable evaluation link, thus solving the problems of static configuration of negotiation strategies and lack of transparency in the evaluation process.

[0027] (4) By planning intelligent agents to dynamically construct a collaborative evaluation system that integrates statistical evaluation indicators, model-based evaluation indicators and large language model-based evaluation indicators, the simple superposition or separation of indicators can be effectively avoided, and the evaluation scheme can be accurately matched with the task characteristics. This solves the problems of static solidification of evaluation indicators and system fragmentation. Attached Figure Description

[0028] Figure 1 This is a flowchart of a multi-agent-based instruction quality evaluation method according to an example of the present invention; Figure 2 This is a block diagram of an electronic device according to an exemplary embodiment of the present invention. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0030] A flowchart of an example of the instruction quality evaluation method based on multi-agent technology of the present invention is shown below. Figure 1 Specifically, it includes: A multi-agent collaborative evaluation framework is constructed, consisting of four types of agents: planning agent, evaluation agent, negotiation agent, and decision-making agent.

[0031] The planning agent is configured to analyze the instruction-input-output triplet, generate instruction metadata, and adaptively generate instruction quality assessment plans based on the instruction metadata.

[0032] Specifically, the planning agent is a large language model with highly efficient parameter fine-tuning: First, based on a standard planning sample set annotated by humans, low-rank adaptation technology is used for fine-tuning, enabling the model to learn and generate dynamic instruction quality assessment plans that conform to a predefined structure. In this stage, the training samples are in the form of... ,in This represents a triple containing instructions, inputs, and outputs. The manually annotated plan is formatted correctly and meets expectations.

[0033] set up To monitor the frozen base model parameters during the fine-tuning phase, The low-rank fitting parameters are used for training during the supervised fine-tuning phase. Therefore, the effective parameters for the supervised fine-tuning phase are... .

[0034] Based on the fine-tuning model, a preference dataset consisting of high-quality and low-quality planning was used. ,in, This represents the current instruction-input-output triplet. This represents the standard program constructed by the t-th person in the preference dataset. Let represent the t-th plan generated by a weak or small model in the preference dataset, where T represents the number of data points in the preference dataset. Direct Preference Optimization (DPO) algorithm is used for preference alignment: Let the trainable low-rank fitness parameters in the DPO stage be . , Can be trained individually or in If training continues on this basis, the effective parameters for the DPO stage are: .

[0035] The loss minimized by direct preference optimization is defined as:

[0036] in, This represents the loss minimized when using the direct preference optimization algorithm for preference alignment. The basic model parameters representing the frozen state of the large language model. This represents the trainable low-rank adaptation parameters during the direct preference optimization phase. Represents a preference dataset, This represents a sample of preference data in the preference dataset. This represents a standard program constructed artificially, including instruction metadata, configuration of evaluation metrics, and negotiation strategies. This indicates a plan generated by a weak or small model, used to compare preference signals. Indicates in Expectation under the given conditions The sigmoid function represents the model's decision. Superior Confidence level, This is a temperature coefficient used to adjust the model's sensitivity to preference differences; in this invention, it is taken as... =0.5, Represents the implicit reward function;

[0037] in, This indicates that the direct preference optimization stage of the large language model selects when the triple is q. The strategy function, These represent the effective parameters in the direct preference optimization phase of the large language model. This indicates that the direct preference optimization stage of the large language model selects when the triple is q. The strategy function, This indicates that during the fine-tuning phase of the large language model, the choice is made when the triple is q. The strategy function, This indicates that during the fine-tuning phase of the large language model, the choice is made when the triple is q. The strategy function, These represent the effective parameters during the fine-tuning phase of the large language model.

[0038] Throughout the training process, the backbone parameters of the large language model are frozen by low-rank adaptation, and only the inserted low-rank decomposition matrix is ​​updated, thereby reducing computational overhead while maintaining the stability of the output structure.

[0039] Specifically, the planning agent determines the task category and subcategory of an instruction based on the instruction-input-output triplet, keyword matching, and semantic reasoning capabilities of a large language model. Simultaneously, based on keywords, named entities, and overall semantics within the instruction, it identifies whether the instruction involves a specific professional domain. If the domain is uncertain, it is defaulted to a general domain. Instruction complexity is determined by the planning agent based on the instruction's input length, logical hierarchy, external knowledge requirements, and reasoning depth. If the task only involves echoing, printing, or copying, and requires no formatting adjustments or content generation other than precise copying, the instruction complexity is determined to be low; otherwise, it is determined to be high. Finally, the instruction metadata is composed of the task category, subcategory, application domain, and instruction complexity.

[0040] The planning agent generates a set of evaluation indicators based on instruction metadata. Simultaneously, it assigns weights to the selected indicators, ensuring that the sum of the weights for statistically based and model-based indicators, as well as the sum of the weights for indicators based on large language models, is 1.0. For each evaluation indicator, the planning agent assigns a corresponding evaluation agent identifier based on its evaluation paradigm, indicating the executing entity for subsequent evaluation stages.

[0041] Among them, the statistical indicators are limited to: text length, lexical diversity, average line length, alphanumeric ratio, stop word ratio, word count, word repetition rate, and character repetition rate; the model-based indicators are limited to: confusion level, instruction followability difficulty, naturalness, coherence, comprehensibility, and authenticity.

[0042] Evaluation metrics based on large language models include, but are not limited to: coverage, completeness, depth, fluency, clarity, coherence, accuracy, factuality, and consistency. When the evaluation requirements of an instruction cannot be met by the above evaluation metrics, the planning agent will generate new metrics based on the instruction content that rely on the large language model for evaluation.

[0043] The planning agent selects a negotiation strategy from four predefined negotiation strategies based on instruction metadata and writes it into the dynamic instruction quality assessment plan. Specifically, for tasks involving only text classification, label judgment, ambiguous restatement, or preference selection, a voting merging strategy is selected; for tasks with large output differences but providing confidence, and involving rating, ranking, preference strength judgment, or requiring probability output, a confidence-weighted strategy is selected; for tasks involving subjective tasks, logical reasoning, or multi-perspective argumentation, a multi-round debate strategy is selected; and for tasks involving fact verification, policy and regulatory Q&A, and scientific and academic knowledge, an evidence retrieval strategy is selected.

[0044] The instruction metadata, evaluation index set, and negotiation strategy are combined to form the instruction quality assessment plan. The results are written to the output file. The evaluation index set includes an index set based on statistics and a model-based index set, as well as an index set based on a large language model. Each index item in the index set based on statistics and a model-based index set contains an index name and weight, and is uniformly associated with the basic evaluation agent. Each index item in the index set based on a large language model contains an index name, weight, and a specific evaluation agent assigned to it. The negotiation strategy includes the negotiation strategy type and the assigned negotiation agent.

[0045] From the raw instruction-input-output triplet Mapped to evaluation plan o= The process can be formalized as a quintuple:

[0046] in, Indicates instruction metadata; Represents a set of evaluation indicators; Indicates the weight of the evaluation indicators; Indicates the selected negotiation strategy; This represents the mapping between indicators and the allocation of evaluation agents; This represents the mapping function for planning agent pairs.

[0047] In one specific implementation of this embodiment, the planning agent receives a set of instruction triplets, where the instruction is "Give me three reasons why basketball is so popular." The input is empty, and the output is "Basketball is popular worldwide for the following reasons: First, the rules of basketball are simple and easy to learn, and people of all ages can understand and participate in it. Second, basketball is a team sport, so playing basketball with friends is endlessly fun and also a way to make new friends. Third, basketball games are exciting and fast-paced, which attracts people all over the world." The quality assessment plan for the instruction generated after parsing by the planning agent includes: task category is brainstorming, subcategory is reason generation, application field is entertainment, and instruction complexity is... The complexity is defined as follows: The statistical and model-based indicator set includes character length (weight 0.2), word count (weight 0.2), lexical richness (weight 0.2), perplexity (0.2), and instruction compliance difficulty (weight 0.2), and is assigned to the basic evaluation agent. The large language model-based indicator set includes coverage (weight 0.25, assigned to the first specific evaluation agent), completeness (weight 0.25, assigned to the first specific evaluation agent), clarity (weight 0.25, assigned to the second specific evaluation agent), and coherence (weight 0.25, assigned to the second specific evaluation agent). The negotiation strategy employs multi-round debate, with a preset maximum of 3 rounds. The above information, in a structured form, constitutes the instruction quality evaluation plan for this task.

[0048] The evaluation agent is configured to calculate and output evaluation results based on the set of evaluation metrics and their corresponding weights in the instruction quality evaluation plan. The evaluation agent includes a basic evaluation agent and a specific evaluation agent. The basic evaluation agent is configured to perform programmable calculations of statistically based and model-based evaluation metrics on the instruction-input-output triple. The specific evaluation agent is configured to guide a large language model through prompting engineering to perform metric scoring based on the large language model. Using statistically based and model-based metrics during the evaluation phase ensures universality, while dynamically introducing large language model-based metrics during the task refinement phase achieves domain-deep evaluation, effectively avoiding simple metric aggregation or fragmentation, and achieving precise matching between the evaluation scheme and task characteristics.

[0049] In this embodiment, the evaluation task is performed by a basic evaluation agent and multiple specific evaluation agents to generate structured evaluation results. The multiple specific evaluation agents specifically include three agents: a content coverage agent, focusing on coverage, completeness, recall, and depth; a language expression agent, focusing on fluency, clarity, coherence, and language style; and a factual accuracy agent, focusing on accuracy, factuality, consistency, and precision.

[0050] First, the instruction quality assessment plan is parsed to extract a set of indicators based on statistics and models, as well as a set of indicators based on a large language model. Then, an indicator result list is initialized to store the score, description, weight, execution time, anomaly flag, and anomaly information of each indicator.

[0051] The basic assessment agent dynamically loads the corresponding indicator calculation module based on the statistical and model-based indicator sets, and calls the calculation function of the module to perform quantitative analysis on the output text to obtain indicator scores, input statistical information, and abnormal states. The indicator score is used as the indicator value, and together with its weight, execution time, and abnormal information specified in the assessment plan, they are organized into a result dictionary object and appended to the indicator result list. If an abnormality occurs during the indicator calculation process, the abnormality flag is set to true, the score is set to 0.0, and the abnormality details are recorded.

[0052] For each indicator in the indicator set based on the large language model, the prompt word generation function is called, taking the instruction-input-output triple, indicator name, and agent assignment information as input, to generate a structured system prompt word; the corresponding large language model is called to perform chained reasoning, and the standardized score conforming to the format "score: X.XX" is parsed from its response; the parsed score, indicator name, and their weight information are organized into a dictionary object and appended to the indicator result list; if the large language model call fails or the score parsing fails, the exception flag is set to true, the score is set to 0.0, and the exception information is recorded.

[0053] The evaluation process can be formalized as follows:

[0054] in, For triples The evaluation results; This represents the evaluation function, with the input being a sample of instructions. And instruction quality assessment planning; Indicates the name of the evaluation indicator. Represents the set of evaluation indicators. Indicator instruction sample In terms of indicators Quantitative scores on the platform Indicators The assigned weights.

[0055] In one specific implementation of this embodiment, based on the statistical and model-based indicator set and the large language model-based indicator set specified by the planning agent in the previously exemplified implementation, the basic evaluation agent and multiple specific evaluation agents output the specific values ​​of the set. The statistical and model-based indicators include: character length 327, word count 66, lexical richness 34, perplexity 20.07, and instruction compliance difficulty 0.0 marked as anomalous. The large language model-based indicators include: coverage 1.0, completeness 0.67, clarity 0.9, and coherence 0.85.

[0056] The negotiation agents are configured to execute negotiation strategies specified by the planning agents based on the evaluation results and output negotiation results. According to the negotiation strategies specified in the instruction quality evaluation plan, multiple negotiation agents execute corresponding negotiation processes, generating negotiation records containing viewpoints, confidence levels, and arguments.

[0057] Specifically, under the voting merging strategy, multiple negotiation agents output their own position labels for each specific evaluation indicator. The position labels include support, neutrality, or opposition, along with confidence level and brief reasons. Then, the distribution of each position is statistically analyzed to form a voting distribution record, and the majority opinion is used as the preliminary conclusion. When there is a tie, the decision is made according to the position priority rule, where support is superior to neutrality, and neutrality is superior to opposition.

[0058] The negotiation outcome under the voting merger strategy is as follows:

[0059] in, This represents the negotiation result under the voting merge strategy, where q represents the instruction-input-output triple. This represents the voting merge strategy function. For the evaluation results, For the first Natural language statements of a negotiating intelligent agent {Support, Neutral, Oppose} indicates the first... Discrete position labels of a negotiating agent; To negotiate the total number of intelligent agents, it is 3 in this embodiment.

[0060] Under the confidence-weighted strategy, each negotiating agent outputs a score (ranging from 0.0 to 1.0), a confidence level (ranging from 0.0 to 1.0), and explanatory text for each specific evaluation indicator. The system calculates the weighted score for that indicator and determines the final stance based on the range of the weighted score: if the weighted score is ≥ 0.7, it is judged as "support"; if 0.4 ≤ weighted score < 0.7, it is judged as "neutral"; if the weighted score < 0.4, it is judged as "oppose".

[0061] The negotiation result under the confidence-weighted strategy is:

[0062] in, This indicates the negotiation result under the confidence-weighted strategy. This represents the confidence-weighted strategy function. express The corresponding confidence level.

[0063] Under the multi-round debate strategy, multiple negotiating agents update their own positions through multiple rounds of interactive reasoning. In each round, they update their own positions based on the previous round's viewpoints of other agents and propose new arguments or rebuttals. After completing a preset number of rounds, they enter the cross-validation phase. Each negotiating agent compares its initial position with its final position to determine whether it has compromised due to the arguments of others, and outputs a validation result including the original position, the final position, and a compromise explanation. Subsequently, based on the cross-validation results of all negotiating agents, it is determined whether a consensus has been reached on each evaluation indicator. If a consensus is reached, a unified position is output. If no consensus is reached, all divergent viewpoints and their complete reasoning chains are retained and handed over to the subsequent decision-making phase for processing.

[0064] The negotiation results under the multi-round debate strategy function include a complete list of rounds of debate to ensure the traceability and auditability of the process. The negotiation results under the multi-round debate strategy are as follows:

[0065] in, This indicates the outcome of negotiations following a multi-round debate strategy. This represents the multi-round debate strategy function. Let R represent the natural language description of the m-th deliberative agent in the i-th round, and let R represent the debate round number.

[0066] Under the evidence retrieval strategy, before generating their opinions, each negotiation agent generates supporting evidence related to the evaluation indicators based on its internal knowledge base, and outputs a score (0.0–1.0), the reliability of the evidence source (0.0–1.0), the evidence text and explanation; the system integrates the outputs of all agents, calculates the average score of each indicator, and determines the final position according to the following threshold rules: a score ≥ 0.7 is "support", 0.4 ≤ score < 0.7 is "neutral", and a score < 0.4 is "oppose".

[0067] The negotiation result under the evidence retrieval strategy is as follows:

[0068] in, This indicates the negotiation result under the evidence retrieval strategy. This represents the evidence retrieval strategy function. Indicates the first The set of evidence cited by a negotiating agent.

[0069] For negotiation strategies with discrete labels as output (such as vote merging, confidence weighting, and evidence retrieval), the final negotiation result is determined by maximizing the score or using a threshold method. For negotiation strategies with complex semantic structures as output (such as multi-round debate), a unified textual stance is generated by merging statements from each round as the final negotiation result. All negotiation results are organized into a unified structured format to form a negotiation record.

[0070] The viewpoints, confidence levels, and arguments of each negotiating agent are organized into a unified structured format to generate a structured negotiation record.

[0071] In one specific implementation of this embodiment, according to the negotiation strategy (multi-round negotiation) specified by the planning agent in the previously exemplified implementation, in the first round of debate during the negotiation phase, the first negotiation agent, based on the evaluation result of a completeness index score of 0.67 output by the aforementioned specific evaluation agent, outputs a position of opposition to the evaluation result. The argument is: "Although the response mentions three core reasons—ease of learning, social attributes, and stimulation—each reason lacks sufficient elaboration. For example, it does not explain the specific simplicity of the rules compared to other sports, does not mention specific social scenarios such as community leagues, and does not cite specific stimulating elements such as last-minute winners, resulting in insufficient depth of argumentation." The confidence level is 0.88. In the same round of debate, the other negotiation agents express supportive or neutral positions on the indicators they are responsible for, and gradually adjust in subsequent rounds of debate and cross-validation, ultimately reaching a consensus on all indicators: coverage and clarity are supported, completeness is opposed, and coherence is neutral. Based on this, the system determines that a consensus has been reached and ends the preset round of debate.

[0072] The decision-making agent is configured to generate the final instruction quality decision score, decision level, and chain reasoning description based on instruction metadata, instruction quality assessment planning, assessment results, and negotiation results.

[0073] Specifically, the original instruction, input, output, category, domain, and instruction complexity fields are first extracted from the instruction quality assessment plan, and then the indicator assessment scores output during the assessment phase and the structured negotiation records generated during the negotiation phase are embedded in sequence.

[0074] The structured negotiation record, depending on the negotiation strategy adopted, is specifically manifested as: the voting distribution and position reasons under the voting merging strategy, each candidate answer and its confidence level under the confidence weighting strategy, the statement records of each round under the multi-round debate strategy, the cross-validation results and the list of unresolved indicators, or the evidence content, information sources and supporting viewpoints under the evidence retrieval strategy. The above information is spliced ​​together according to the preset prompt template to form the contextual prompt text for the large language model to reason.

[0075] The large language model is invoked, and its output is forced to be structured through a predefined function call interface. The function call interface requires the output to include the following three fields: decision score (range 0.0–1.0), decision level label, and chain reasoning basis. The chain reasoning basis is a piece of natural language text that requires explicit citation of key information from the negotiation stage, including but not limited to the focus of the dispute, core indicator scores, supporting or contradictory evidence fragments, and the evolution of viewpoints in multiple rounds of debate, to ensure that the judgment process is interpretable and auditable.

[0076] The decision score, decision level label, and chain reasoning basis are written into the corresponding data object in the instruction quality assessment plan, and a timestamp is added to identify the time of judgment completion. At the same time, the complete reference path of the intermediate results of the basic assessment, specific assessment, and negotiation stages is retained in the data object to ensure that any judgment conclusion can be traced back to its original source of evidence.

[0077] Decision-making agents use decision functions Based on the negotiation results The resulting decision can be represented as:

[0078] in, , indicating the quality score of the judgment; Indicates the quality level label of the instruction; Expressing the rationale for a decision in natural language.

[0079] In one specific implementation of this embodiment, the decision agent receives the planning phase output (including instruction metadata, evaluation plan, indicator set and negotiation strategy), evaluation phase output and negotiation consensus phase output (all indicators are agreed upon) from the previously exemplified implementation. It outputs a decision score of 0.82, a quality label of "good", and chain reasoning basis in text form. It emphasizes the clarity of the instructions and the comprehensiveness of the output coverage, but also points out the problem of superficial argumentation caused by insufficient completeness. Finally, it is considered that its overall quality is good and suitable for training models for structured reasoning.

[0080] Write the updated complete instruction quality assessment plan into the preset output path, record the processing log synchronously, and store the entire process assessment record, consisting of the instruction quality assessment plan, basic assessment results, specific assessment results, negotiation conclusion report, and final judgment result, into the memory module.

[0081] In one exemplary embodiment, a computer-readable storage medium is included, which stores a computer program that, when executed by a processor, implements the method described above.

[0082] Please see Figure 2 In one exemplary embodiment, the device further includes an electronic device including at least one processor, at least one memory, and at least one communication bus.

[0083] The memory stores a computer program, which includes computer-readable instructions. The processor calls the computer-readable instructions stored in the memory through a communication bus to execute the above method.

[0084] In one exemplary embodiment, a computer program product is proposed, including a computer program / instructions that, when executed by a processor, implement the method described above.

[0085] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for evaluating instruction quality based on multi-agent systems, characterized in that, Includes the following steps: Construct a multi-agent collaborative evaluation framework consisting of four types of agents: planning agent, evaluation agent, negotiation agent, and decision-making agent; The planning agent analyzes the instruction-input-output triplet, generates instruction metadata, and adaptively generates instruction quality assessment planning based on the instruction metadata; The evaluation agent calculates and outputs the evaluation results based on the set of evaluation indicators and the weights of each indicator in the instruction quality evaluation plan. Based on the evaluation results, the negotiating agent executes the negotiation strategy specified by the planning agent and outputs the negotiation results; Based on the instruction metadata, instruction quality assessment plan, assessment results, and negotiation results, the decision-making agent generates the final instruction quality decision score, decision level, and chain reasoning explanation.

2. The instruction quality evaluation method based on multi-agent systems according to claim 1, characterized in that, The instruction metadata includes: task category, subcategory, application area, and instruction complexity.

3. The instruction quality evaluation method based on multi-agent systems according to claim 1, characterized in that, The instruction quality assessment plan includes: a set of assessment indicators and their weights, negotiation strategy types, and allocation information for corresponding execution agents.

4. The instruction quality evaluation method based on multi-agent systems according to claim 3, characterized in that, Negotiation strategies include: When the instruction involves text classification, label determination, ambiguous restatement, or preference selection, a voting merging strategy is adopted; the negotiation result under the voting merging strategy is: in, This represents the negotiation result under the voting merge strategy, where q represents the instruction-input-output triple. This represents the voting merge strategy function. For the evaluation results, For the first Natural language statements of a negotiating intelligent agent {Support, Neutral, Oppose} indicates the first... Discrete position labels of a negotiating agent; To negotiate the total number of intelligent agents; When the instruction involves rating, ranking, preference strength judgment, or requires probability output, a confidence-weighted strategy is adopted; the negotiation result under the confidence-weighted strategy is: in, This indicates the negotiation result under the confidence-weighted strategy. This represents the confidence-weighted strategy function. express The corresponding confidence level; When the instructions involve subjective tasks, logical reasoning, or multi-perspective arguments, a multi-round debate strategy is adopted; the negotiation results under the multi-round debate strategy are as follows: in, This indicates the outcome of negotiations following a multi-round debate strategy. This represents the multi-round debate strategy function. Let R represent the natural language description of the m-th deliberative agent in the i-th round, and let R represent the debate round number. When the instruction involves fact-checking, policy and regulatory Q&A, or scientific and academic knowledge, an evidence retrieval strategy is adopted. The negotiation result under the evidence retrieval strategy is as follows: in, This indicates the negotiation result under the evidence retrieval strategy. This represents the evidence retrieval strategy function. Indicates the first The set of evidence cited by a negotiating agent.

5. The instruction quality evaluation method based on multi-agent systems according to claim 1, characterized in that, The evaluation agent includes a basic evaluation agent and a specific evaluation agent. The basic evaluation agent is configured to perform programmable computation of statistical and model-based evaluation metrics on instruction-input-output triples. The specific evaluation agent is configured to guide the large language model to score metrics based on the large language model through prompting engineering.

6. The instruction quality evaluation method based on multi-agent systems according to claim 1, characterized in that, The planning agent is a fine-tuning large language model. First, based on a standard planning sample set with manual annotation, low-rank adaptation technology is used for fine-tuning, so that the large language model learns to generate dynamic instruction quality assessment planning that conforms to a predefined structure. The training samples are in the form of Where q represents the instruction-input-output triple, These are standard planning samples with manual annotations; Then, a preference dataset is constructed, and preference alignment is performed using the direct preference optimization algorithm.

7. The instruction quality evaluation method based on multi-agent systems according to claim 6, characterized in that, The loss for minimizing preference alignment using the direct preference optimization algorithm is defined as: in, This represents the loss minimized when using the direct preference optimization algorithm for preference alignment. The basic model parameters representing the frozen state of the large language model. This represents the trainable low-rank adaptation parameters during the direct preference optimization phase. Represents a preference dataset, This represents a sample of preference data in the preference dataset. Standard planning representing artificial structures, This represents the plan generated by the model. Indicates in Expectation under the given conditions The sigmoid function represents the model's decision. Superior Confidence level, This is a temperature coefficient used to adjust the model's sensitivity to differences in preferences. Represents the implicit reward function; in, This indicates that the direct preference optimization stage of the large language model selects when the triple is q. The strategy function, These represent the effective parameters in the direct preference optimization phase of the large language model. This indicates that the direct preference optimization stage of the large language model selects when the triple is q. The strategy function, This indicates that during the fine-tuning phase of the large language model, the choice is made when the triple is q. The strategy function, This indicates that during the fine-tuning phase of the large language model, the choice is made when the triple is q. The strategy function, These represent the effective parameters during the fine-tuning phase of the large language model.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

9. An electronic device, characterized in that, The device includes a processor and a memory, the processor being interconnected with the memory, wherein the memory is used to store a computer program, the computer program including computer-readable instructions, and the processor is configured to invoke the computer-readable instructions to perform the method as described in any one of claims 1 to 7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the method described in any one of claims 1 to 7.

Images