Intelligent assessment method, medium, and device based on bias elimination and feature separation

Abstract

This invention discloses an intelligent evaluation method, medium, and device based on bias elimination and feature separation, relating to the field of artificial intelligence technology. The method includes: receiving user queries and generating a candidate solution set; scheduling multiple heterogeneous agents to form an evaluation matrix; each agent comparing candidate solutions pairwise to construct an observation dataset; constructing a parameterized model based on this dataset, solving for optimal parameters by minimizing a loss function containing preference constraints, whereby the constraints are used to suppress the model's systematic preference for specific latent semantic features in candidate solutions, thereby eliminating evaluation bias; calculating and ranking the global preference scores of candidate solutions based on the optimal parameters, and simultaneously evaluating their credibility and dynamically adjusting their weights according to the deviation of each agent's preference features from the consensus. This invention improves the objectivity and reliability of multi-agent evaluation by automatically separating and suppressing non-core feature preferences through modeling, and reduces reliance on manual calibration.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and specifically to an intelligent evaluation method, medium, and device based on bias elimination and feature separation. Background Technology

[0002] With the rapid development of generative artificial intelligence, large language models (LLMs) are widely used in scenarios such as text generation, code writing, and complex logical reasoning. To evaluate the quality of the model's generated results, the industry has gradually developed an automated evaluation technique called "LLM-as-a-judge." This technique typically employs pairwise comparative judgments, allowing one or more models to compare and score two candidate results. In industrial scenarios requiring high reliability, multiple different models are often introduced to form a "jury," and the final ranking is output by aggregating the opinions of multiple evaluators.

[0003] However, existing evaluation methods based on multi-model collaboration have the following significant drawbacks:

[0004] Evaluation bias is difficult to eliminate: Different evaluation models (agents) have inherent differences in training data, architecture, and prompt word strategies, leading to systematic preferences for non-core features of candidate solutions (such as specific expression styles, text complexity, etc.). This bias contaminates the evaluation results, making the final ranking not entirely based on the core qualities of the solutions (such as accuracy and completeness), thereby reducing the objectivity and reliability of the evaluation.

[0005] Unreasonable weighting of evaluators: Existing systems often assign equal weights to all participating models or use simple majority voting, ignoring the issue that different models contribute differently to the overall evaluation results due to their own biases and differences in capabilities. A model with strong biases towards non-core features can inappropriately influence the final consensus.

[0006] Reliance on manual calibration: To correct the aforementioned biases, existing methods often require the introduction of large amounts of high-quality manually labeled data to calibrate or fine-tune the evaluation model. This approach is costly, difficult to adapt to dynamically changing tasks and emerging new models, and lacks scalability and timeliness. Summary of the Invention

[0007] In view of the above problems, the present invention provides a technical solution for intelligent evaluation based on bias elimination and feature separation, which solves the problem of evaluation result bias caused by the systematic preference of different evaluation models for non-core content features (such as expression style, text length, etc.) in the existing "LLM-as-a-judge" multi-model collaborative evaluation method, as well as the problems of insufficient evaluation reliability and high cost of relying on manual calibration.

[0008] To achieve the above objectives, in a first aspect, the present invention provides an intelligent evaluation method based on bias elimination and feature separation, the method comprising:

[0009] S1: Receive a natural language query input by the user, and generate a set of M candidate solutions based on the natural language query. And schedule K heterogeneous agents to form an evaluation matrix. The heterogeneous intelligent agents differ in model architecture, parameter configuration, or evaluation strategy.

[0010] S2: For each agent, schedule the agent to perform pairwise comparisons of candidate solutions in the candidate solution set S. For any two candidate solutions S i and S j By the kth intelligent agent Determine and generate indicator variables When the intelligent agent Determine candidate solution S i It outperforms candidate solution S in the preset evaluation dimensions. j hour, =1, otherwise =0, the preset evaluation dimensions include the accuracy, completeness, and relevance to the natural language query of the candidate solutions, and all comparison results are collected to construct an observation dataset. ={(S i ,S j , )|S i ,S j ∈S, i≠j};

[0011] S3: Based on observation dataset A parameterized model is constructed to characterize the preference features of each agent, and the optimal parameter solution set of the parameterized model is solved by minimizing the loss function containing preference constraints. The preference constraints are used to constrain the overall preference strength of the model parameters for a specific feature dimension in order to eliminate systematic bias in the evaluation process. The specific feature dimension is a latent semantic feature dimension obtained by learning the representation of the content of the candidate scheme.

[0012] S4: Based on the optimal parameter solution set, calculate the global preference score of each candidate scheme and sort and output it. According to the degree of deviation between the preference feature vector of each agent and the consensus preference vector reflected in the optimal parameter solution set, evaluate the credibility of the agent and adjust the dynamic weight.

[0013] Furthermore, in step S3, constructing a parameterized model to characterize the preference features of each agent includes:

[0014] Each candidate solution is vectorized into intrinsic attributes of two dimensions: core accuracy and information divergence. Here, candidate solution S... i Core accuracy is denoted as Candidate Solution S i Information divergence is denoted as ;

[0015] For each intelligent agent Configure discrimination parameters and preference parameters ;

[0016] Building intelligent agents For candidate solution S i Comprehensive perceived utility function The formula is as follows:

[0017] ;

[0018] intelligent agent Determine candidate solution S i It outperforms candidate solution S in the preset evaluation dimensions. j probability distribution function as follows:

[0019] ;

[0020] The loss function that includes preference constraints is denoted as The calculation formula is as follows:

[0021] ;

[0022] Where K represents the total number of agents, and k represents the index of the agent. This represents a hyperparameter used to control the weights of preference constraint terms. This represents a preference constraint term, used to force the overall preference of the agent group for information divergence characteristics to tend towards neutrality.

[0023] Furthermore, the credibility assessment and dynamic weight adjustment of the agent include:

[0024] The discrimination parameters of each agent obtained from the optimal parameter solution set and preference parameters An assessment will be conducted, as follows:

[0025] When the intelligent agent Discrimination parameters When the value falls below a first preset threshold, a circuit breaker mechanism is triggered to shut down the intelligent agent. Remove from evaluation matrix A;

[0026] When the intelligent agent Preference parameters | When the threshold is exceeded, the agent's power level is reduced. Weighting in subsequent evaluation tasks.

[0027] Furthermore, the differences in model architecture, parameter configuration, or evaluation strategy of the heterogeneous intelligent agents are achieved through at least one of the following methods:

[0028] In terms of model architecture, large language models with different network structures, parameter counts, or training datasets are used as the base models for each agent.

[0029] In terms of evaluation strategy, each agent is configured with prompt word templates that focus on different evaluation dimensions or have different logical structures;

[0030] In terms of parameter configuration, different sampling temperature parameters, top-p parameters, or output length limit parameters are set for each agent.

[0031] Furthermore, in step S3, the parameterized model is constructed by modeling the specific feature dimension as a set of confusion variables to be controlled. The specific implementation steps are as follows:

[0032] Decouple the candidate solutions by features, and decouple each candidate solution S i Represented as the decoupled feature pair (X) i Z i ), where X i Z is the core quality feature vector, used to characterize the intrinsic attributes directly related to the accuracy and completeness in the preset evaluation dimensions; i To obfuscate the feature vector, which is used to characterize the latent semantic features extracted from the content of the candidate scheme that constitute the specific feature dimension;

[0033] Based on the observation dataset D k With the decoupled feature pair (X) i Z i A structural causal model is constructed, and the model parameters of the structural causal model are solved by minimizing the composite loss function to estimate the core quality feature vector X. iThe average causal effect of the difference in the agent’s pairwise comparison preference judgments;

[0034] The composite loss function L causal The representation is as follows:

[0035] L causal =L fit +β·L balance ;

[0036] Among them, L fit This is a data fitting term used to correlate the probability predicted by the structural causal model with the indicator variable. Consistent; L balance This is a confusion balancing term used to weaken the confusion in the feature vector Z during the optimization process. i With core quality feature vector X i The statistical dependence between them is used to simulate randomized controlled trials and control the confusion feature vector {Z} of all candidate protocols. i The influence of latent semantic features represented by}, where β is a balancing hyperparameter;

[0037] In step S4, based on the optimal parameter solution set, the global preference score of each candidate scheme is calculated and sorted, including:

[0038] Based on the solved structural causal model, each candidate solution S i The confusion feature vector Z i The intervention is a pre-defined baseline vector Z0, while maintaining the candidate solution S. i Core quality feature vector X i Under the same counterfactual condition, calculate the candidate scheme S determined by all agents. i The average predicted probability that outperforms any other candidate solution is used as the candidate solution S. i The final global preference score.

[0039] Furthermore, the global preference score for each candidate solution is calculated and ranked, and the output includes:

[0040] Based on the optimal parameter solution set, each agent The preference parameters for each agent Define an independent evaluation function F k (·);

[0041] For each candidate solution S i The candidate solution S i The content features are input into all K evaluation functions {F k} Calculate K scalar output values ​​respectively, and arrange these K scalar output values ​​in order to form scheme S.i K-dimensional utility vector V i The calculation formula is as follows:

[0042] V i =(F1(S i ), F2(S i ),...,F K (S i ), where the k-th component F k (S i S is a candidate solution. i The independent utility value under the evaluation function defined by agent k;

[0043] The K-dimensional utility vector {V} of all candidate solutions i} is mapped to a K-dimensional target space with K agents as orthogonal basis vectors, and each candidate solution S i This corresponds to a coordinate point in the space;

[0044] In the K-dimensional target space, the non-dominated solution set is selected according to the Pareto dominance criterion, specifically including: for a candidate solution S i If there is no other candidate solution S j This makes S j utility vector V j It is greater than or equal to S in every dimension k. i utility vector V i The corresponding component of , and there exists at least one dimension k' such that ( > ( If ) is true, then S is determined to be true. i For a Pareto nondominated solution, the set of all Pareto nondominated solutions constitutes the Pareto optimal solution set.

[0045] Output the Pareto optimal solution set, and for each candidate solution S in the Pareto optimal solution set... i Perform the following steps to generate a report on its strengths:

[0046] Iterate through each evaluation dimension k and check if, within the Pareto optimal solution set, except for S... i Besides itself, is there another candidate solution S? j Its utility value F on that dimension k k (S j )>F k (S i );

[0047] There will be no such candidate solution S. jThe indices of dimension k are collected and recorded as an index set, and the index set is used as a candidate scheme S. i The advantage dimension report is output together with the Pareto optimal solution set.

[0048] Furthermore, a model robustness enhancement stage is included before step S3, which includes the following steps:

[0049] Construct and train an adversarial sample generator, the adversarial sample generator using any candidate scheme S from the candidate scheme set S. i The original content is the input, and the output is an adversarial candidate solution S. advi The training objective of the generator is as follows: to make S advi With S i Maintain consistency across the core quality feature vectors defined by the parameterized model, while maximizing S. advi Feature strength in the latent semantic feature dimension;

[0050] Using the trained adversarial sample generator, an adversarial variant is generated for each scheme in the candidate scheme set S. All adversarial variants are then added to the candidate scheme set S to form an augmented candidate scheme set S. aug Based on S aug Then repeat step S2 to construct an augmented observation dataset containing the results of adversarial sample comparisons;

[0051] Step S3 is performed using the augmented observation dataset, and the loss function further includes an adversarial regularization term L. adv The L adv It is constructed as follows: for each pair of original candidate schemes S i Its corresponding adversarial candidate solution S advi The parameterized model is calculated as S. advi With S i Find the norm of the difference between the assigned core quality scores and minimize the sum.

[0052] Furthermore, the credibility assessment and dynamic weight adjustment of the agent also include:

[0053] Based on the trained parameterized evaluation model, the calculation of each agent... The preference feature vector is calculated, and the cosine similarity between the preference feature vector and the consensus preference vector of all agents is calculated to obtain the consensus deviation δ. k Simultaneously, computational intelligent agents In the observation dataset D k Internal consistency score κ of the judgment results k If the intelligent agent Simultaneously satisfying δ k >τ1 and κ k >τ2, then the agent Identify as a highly consistent dissenter, where τ1 is a preset first score threshold and τ2 is a preset second score threshold;

[0054] Cluster analysis was performed on all preference feature vectors identified as having high consistency and dissent to identify a set of common preference patterns that significantly deviate from the consensus {P1, P2, ..., P...}. m}, where m is the total number of preference patterns;

[0055] For each identified preference pattern P t The value of t ranges from [1, m]. Perform the following operations: construct a positive sample set from the winning candidate solutions preferred by all dissenting agents belonging to this preference pattern, and construct a negative sample set from their corresponding losing solutions; extract a set of features that can best explain the preference pattern P from the original content features of the positive and negative sample sets using feature importance analysis or gradient inversion methods. t Key content feature set {F t};

[0056] Key content features F t Input a pre-trained natural language generation model, and drive the pre-trained natural language generation model to synthesize a segment describing the key content features F. t The natural language definition text of the potential evaluation perspective or principle represented, and the natural language definition text is used as an evaluation dimension to expand the proposal output.

[0057] In a second aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the intelligent evaluation method based on bias elimination and feature separation as described in the first aspect of the present invention.

[0058] In a third aspect, the present invention provides an electronic device having a computer program stored thereon, including a processor and a storage medium, wherein the computer program is stored on the storage medium, and when executed by the processor, the computer program implements the intelligent evaluation method based on bias elimination and feature separation as described in the first aspect of the present invention.

[0059] Unlike existing technologies, the above-mentioned technical solution involves an intelligent evaluation method, medium, and device based on bias elimination and feature separation, belonging to the field of artificial intelligence technology. The method includes: receiving user queries and generating a candidate solution set; scheduling multiple heterogeneous agents to form an evaluation matrix; each agent comparing candidate solutions pairwise to construct an observation dataset; constructing a parameterized model based on this dataset; solving for optimal parameters by minimizing a loss function containing preference constraints, whereby the constraints are used to suppress the model's systematic preference for specific latent semantic features in candidate solutions, thereby eliminating evaluation bias; calculating and ranking the global preference scores of candidate solutions based on the optimal parameters; and simultaneously evaluating the credibility and dynamically adjusting the weights of each agent based on the deviation of its preference features from the consensus. This invention improves the objectivity and reliability of multi-agent evaluation by automatically separating and suppressing non-core feature preferences through modeling, and reduces reliance on manual calibration.

[0060] The above description of the invention is merely an overview of the technical solution of the present invention. In order to enable those skilled in the art to better understand the technical solution of the present invention and to implement it based on the description and drawings, and to make the above-mentioned objectives and other objectives, features and advantages of the present invention easier to understand, the following description is provided in conjunction with the specific embodiments and drawings of the present invention. Attached Figure Description

[0061] The accompanying drawings are only used to illustrate the principles, implementation methods, applications, features, and effects of specific embodiments of the present invention and other related contents, and should not be considered as limitations on the present invention.

[0062] In the accompanying drawings of the instruction manual:

[0063] Figure 1 The flowchart illustrates the intelligent evaluation method based on bias elimination and feature separation as described in the first exemplary embodiment of the present invention.

[0064] Figure 2 This is a schematic diagram of an electronic device according to an exemplary embodiment of the present invention;

[0065] The reference numerals used in the above figures are explained as follows:

[0066] 10. Electronic equipment; 101. Processor; 102. Storage medium. Detailed Implementation

[0067] To explain in detail the possible application scenarios, technical principles, specific feasible solutions, and the objectives and effects that can be achieved by this invention, the following detailed description is provided in conjunction with the listed specific embodiments and accompanying drawings. The embodiments described herein are only used to more clearly illustrate the technical solutions of this invention, and are therefore only examples, and should not be used to limit the scope of protection of this invention.

[0068] In the first aspect, such as Figure 1 As shown, this invention provides an intelligent evaluation method based on bias elimination and feature separation, the method comprising:

[0069] S1: Receive a natural language query input by the user, generate a set of M candidate solutions based on the natural language query, and schedule K heterogeneous intelligent agents to form an evaluation matrix, wherein the heterogeneous intelligent agents have differences in model architecture, parameter configuration or evaluation strategy;

[0070] S2: For each agent, schedule the agent to perform pairwise comparisons of candidate solutions in the candidate solution set S. For any two candidate solutions S i and S j By the kth intelligent agent Determine and generate indicator variables When the intelligent agent Determine candidate solution S i It outperforms candidate solution S in the preset evaluation dimensions. j hour, =1, otherwise =0, the preset evaluation dimensions include the accuracy, completeness, and relevance to the natural language query of the candidate solutions, and all comparison results are collected to construct an observation dataset. ={(S i ,S j , )|S i ,S j ∈S, i≠j};

[0071] S3: Based on observation dataset A parameterized model is constructed to characterize the preference features of each agent, and the optimal parameter solution set of the parameterized model is solved by minimizing the loss function containing preference constraints. The preference constraints are used to constrain the overall preference strength of the model parameters for a specific feature dimension in order to eliminate systematic bias in the evaluation process. The specific feature dimension is a latent semantic feature dimension obtained by learning the representation of the content of the candidate scheme.

[0072] S4: Based on the optimal parameter solution set, calculate the global preference score of each candidate scheme and sort and output it. According to the degree of deviation between the preference feature vector of each agent and the consensus preference vector reflected in the optimal parameter solution set, evaluate the credibility of the agent and adjust the dynamic weight.

[0073] In this embodiment, bias elimination refers to removing systematic preference biases caused by non-core quality factors such as text length, information redundancy, writing style, and level of detail from the evaluation behavior of large model judges, so that the evaluation results are determined only by the correctness, compliance, and accuracy of the solution itself, eliminating the interference of superficial features on the final ranking.

[0074] Feature separation refers to splitting the intrinsic core quality and surface interference features of a candidate solution into two sets of independent and non-interfering feature variables, achieving a decoupled representation of "quality as quality and style as style", providing an underlying foundation for subsequent bias stripping.

[0075] Heterogeneous intelligent agents refer to AI evaluation agents composed of multiple underlying base models, different prompt words, and different parameter configurations. Their heterogeneity ensures a diverse evaluation perspective and avoids common biases and evaluation blind spots caused by a single model.

[0076] Indicator variables It refers to the digitization of the pairwise comparison results between candidate scheme i and candidate scheme j by the k-th agent. It is the only observation signal in the entire unsupervised modeling process, requiring no manual annotation or expert benchmark answer.

[0077] Observation dataset D k The data, consisting of the pairwise comparisons of all agents with all candidate solutions, forms the basis for joint optimization, parameter estimation, and bias removal of the model.

[0078] Preference constraint terms refer to regularization constraints embedded in the loss function, which are used to forcibly suppress the collective preference of the agent group for a certain type of surface features (such as length, redundant information, and level of detail), thereby eliminating systematic bias at its source.

[0079] Systematic bias refers to common industry defects that exist in automated evaluation of large model judges, including length bias (preferring long texts), circular bias (A>B, B>C, C>A), consistency bias, random judgment, and other problems.

[0080] The consensus preference vector is a neutral and objective evaluation tendency vector that is learned by the evaluation behavior of all agents. It is used to measure whether an individual agent deviates from the objective standard or whether there is an extreme preference.

[0081] The global preference score is an objective score determined solely by the core quality of the candidate solution after stripping away all style biases and interfering features. It is used for final ranking and output of the optimal solution.

[0082] In step S1, the system first receives a natural language query from the end user. This query is typically a text with strong compliance and professional requirements, such as financial policy consultation, reimbursement rule inquiries, tax standard confirmation, or medical insurance policy interpretation. Based on the query, the system invokes different generation models, generation strategies, or subsystems to generate M candidate solutions with differences in content, detail, and structure, forming a candidate solution set S. Simultaneously, the system schedules K heterogeneous agents to form an evaluation matrix A. These agents have clear differences in their underlying big model, prompts, and parameter configurations, ensuring diversity and coverage of evaluation perspectives and preventing bias from a single model from contaminating the overall result.

[0083] In step S2, the system drives each agent in the evaluation matrix to perform a full pairwise comparison of all candidate solutions in the set. For any pair of candidate solutions S... i and candidate solutions S j The k-th agent makes a judgment on the merits of the evaluation based on preset evaluation dimensions (accuracy, completeness, query relevance) and uses a binary indicator variable. Record the results: Determine candidate solution S i A better solution is recorded as 1, otherwise as 0. The system iterates through all agents and all candidate solution pairs, compiling all comparison results into an observation dataset D. k Step S2 generates self-supervised signals solely through cross-comparison between multiple agents, completely independent of manual annotation and expert benchmark answers, thus reducing the reliance on calibration costs from the outset.

[0084] In step S3, the system is based on the constructed observation dataset D k A parameterized model is constructed to accurately characterize the judgment ability, style preference, and behavioral features of each agent. This model decomposes the features of candidate solutions into core quality features and surface interference features, and assigns learnable parameters to each agent to describe its discrimination strength and style tendency. To fundamentally eliminate systematic biases inherent in large models, such as length bias and redundant preferences, a preference constraint term is specifically added to the loss function. This constraint term mathematically forces all agents to maintain a neutral overall preference for surface features, preventing the evaluation group from being biased by a particular type of non-core feature. By minimizing this constrained loss function, the system can solve for a set of optimal parameters that fully preserve core quality information while maximizing the removal of bias interference.

[0085] In step S4, based on the optimal parameter set, the system calculates a global preference score for each candidate solution, unaffected by style, length, or redundant information, and sorts them from highest to lowest score, outputting the solution that best meets the core quality requirements. Simultaneously, based on the preference characteristics of each agent reflected in the optimal parameters, the system calculates the degree of deviation between the agent's preference and the group consensus preference vector, and uses this as a basis for credibility rating of the agents: agents with excessive deviation, random judgments, or extreme preferences will be dynamically downgraded or eliminated, thereby achieving self-optimization and continuous stability of the evaluation system.

[0086] The method in this embodiment has the following beneficial effects:

[0087] First, it completely eliminates the reliance on manual annotation and expert calibration. Traditional evaluation methods must rely on a large amount of high-quality manually annotated data for temperature scaling, model fine-tuning, or RLHF alignment. In scenarios with frequent dynamic updates, such as finance, medical insurance, and policies, the annotation cost is extremely high and the timeliness is extremely poor. This invention relies entirely on multi-agent self-supervised comparison signals to complete modeling, achieving automated evaluation and bias elimination with zero human intervention and zero expert involvement, significantly reducing system operation and maintenance costs and iteration thresholds.

[0088] Second, it achieves separation of core quality from superficial features at the underlying architecture level. Existing technologies generally conflate solution quality with text style, length, and redundant information, making evaluation results easily misled by "lengthy but erroneous, comprehensive but misleading" content. This invention, through a feature separation architecture, strictly decouples core quality from interfering features, allowing evaluation to return to its essence of "correctness, compliance, and accuracy," significantly improving the reliability of evaluations in demanding scenarios such as finance and policy.

[0089] Third, it possesses dynamic self-governance and security control capabilities. Traditional multi-model evaluation uses static mechanisms such as simple average voting and weighted voting, which cannot identify and eliminate agents that make random, biased, or incorrect judgments. This invention achieves automatic identification, dynamic weight reduction, and even circuit breaker removal of untrusted agents by calculating the deviation between agents and consensus preferences in real time, ensuring that the evaluation system remains stable, objective, and controllable.

[0090] Fourth, it is highly adaptable to scenarios with strong regulatory oversight and high compliance requirements. In fields such as hospitals, public institutions, and government finance, AI assessment results must be interpretable, traceable, and verifiable. This invention, through parametric modeling and preference quantification, ensures that every assessment, every score, and every weight adjustment has a clear mathematical basis, fully meeting the stringent regulatory requirements of the public service sector for AI systems to be "explainable, controllable, and auditable."

[0091] In some embodiments, in step S3, constructing a parameterized model for characterizing the preference features of each agent includes:

[0092] Each candidate solution is vectorized into intrinsic attributes of two dimensions: core accuracy and information divergence. Here, candidate solution S... i Core accuracy is denoted as Candidate Solution S i Information divergence is denoted as ;

[0093] For each intelligent agent Configure discrimination parameters and preference parameters ;

[0094] Building intelligent agents For candidate solution S i Comprehensive perceived utility function The formula is as follows:

[0095] ;

[0096] intelligent agent Determine candidate solution S i It outperforms candidate solution S in the preset evaluation dimensions. j probability distribution function as follows:

[0097] ;

[0098] The loss function that includes preference constraints is denoted as The calculation formula is as follows:

[0099] ;

[0100] Where K represents the total number of agents, and k represents the index of the agent. This represents a hyperparameter used to control the weights of preference constraint terms. This represents a preference constraint term, used to force the overall preference of the agent group for information divergence characteristics to tend towards neutrality.

[0101] In this embodiment, core accuracy This refers to the factual accuracy, standard matching, and policy compliance of the candidate solution in terms of professional content such as financial policies, tax rules, medical insurance clauses, and business compliance. It is the only core intrinsic attribute that needs to be retained in the evaluation system of this invention, and it directly determines whether the candidate solution is usable and compliant.

[0102] Information divergence This refers to the richness of non-core information contained in the candidate solution, such as extended descriptions, process extensions, irrelevant details, redundant statements, and additional examples. It belongs to the superficial style interference feature and has no necessary correlation with the correctness of the candidate solution itself.

[0103] Discrimination parameters This is used to quantify the objective sensitivity and judgment ability of the k-th agent in distinguishing whether a candidate solution is right or wrong, superior or inferior, or compliant. The larger the value, the more stable and accurate the agent's judgments. The smaller the value, the more random and chaotic the judgment is.

[0104] Preference parameters This is used to quantify the intensity of the k-th agent's subjective preference for information dispersion (text length, content detail, redundancy). >0 indicates that the agent prefers detailed, comprehensive, and long text answers; <0 indicates a preference for concise, straightforward, and short answers; ≈0 indicates neutrality and no preference.

[0105] Overall perceived utility It refers to the agent's subjective overall score for candidate solutions, which is composed of the objective core accuracy and subjective style preference, and fully reflects the agent's true evaluation behavior.

[0106] Preference constraints It refers to the L2 regularization penalty term on the preference parameters of all agents. Its function is to force the overall preference of all agents for information divergence to approach 0 through mathematical constraints, so as to achieve group evaluation neutrality and eliminate collective length bias.

[0107] The Exp-BT two-dimensional extended probability model is an extended model innovatively proposed in this invention based on the traditional Bradley-Terry model, which supports dual-feature decoupling, agent-specific parameters, and unsupervised bias removal.

[0108] The working principle of this embodiment is as follows:

[0109] First, the system uses a two-dimensional vector representation for each candidate solution, decomposing the unobservable intrinsic quality into two independent variables: core accuracy. Information divergence .in The ultimate goal of the evaluation is to demonstrate the professional correctness and compliance of the candidate solutions. The level of detail, redundancy, and expansion of candidate solutions represent interference factors that need to be removed. This step completes feature separation at the representation layer.

[0110] Secondly, the system provides each agent a with... k Assign two personalized learnable parameters: discrimination parameter With preference parameters . Characterizing the judgment ability of an intelligent agent It characterizes their style preference regarding the degree of information dissemination. Using these two parameters, the system can accurately determine the evaluation behavior patterns of each large-scale model referee.

[0111] Based on this, the system constructs a comprehensive perception utility function of the agent regarding the solution: This formula clearly shows that: the agent's subjective rating = objective core quality + personal style preference, thus achieving mathematical decoupling of ability and preference.

[0112] Subsequently, the system constructs a pairwise probabilistic evaluation model based on the logistic distribution, transforming utility differences into probabilistic outputs: After substituting the utility function, the model can be directly derived from... Computational agent to determine candidate scheme S i Superior to candidate solution S j The probability of winning or losing. This model breaks through the limitation of the traditional BT model, which can only handle one-dimensional wins and losses, and can simultaneously fit the combined influence of core quality and style preference.

[0113] To eliminate systematic bias under unsupervised conditions, this invention innovatively adds a preference-neutral regularization term to the standard maximum likelihood loss function, forming the final loss function. The first part of the loss function is the negative log-likelihood loss, which ensures that the model's predictions are consistent with the observed data; the second part is the preference constraint term, which penalizes excessively large... This forces all intelligent agents to remain neutral regarding the degree of information dispersion.

[0114] The system employs the Alternating Directional Multiplier Method (ADMM) or gradient descent algorithm to minimize the loss function, ultimately converging to obtain the optimal parameter set { This process requires no manual annotation, no expert benchmarks, and no temperature scaling; it achieves bias elimination and parameter calibration solely through multi-agent self-supervised signals.

[0115] This embodiment has the following beneficial effects:

[0116] First, it breaks through the fundamental limitation of traditional evaluation models, which can only perform one-dimensional modeling. Traditional Bradley-Terry models, Elo rating models, and the like can only handle the one-dimensional relationship of "win / loss" and cannot distinguish between "good quality" and "style preference". This invention is the first to achieve two-dimensional modeling of core accuracy and information divergence in pairwise comparison evaluation, and completes the decoupling of quality and style from the mathematical foundation.

[0117] Second, it achieves precise removal of length bias in an unsupervised manner. Length bias is the most stubborn and common defect in large model referees, and existing technologies can only alleviate it through manual annotation. This invention automatically neutralizes group bias with zero manual cost by using a preference-neutral regularization term, restoring the pure core accuracy θ, which is unaffected by length, detail, or redundancy, thus returning the evaluation results to their professional essence.

[0118] Third, achieve precise modeling of each intelligent agent that is quantifiable, interpretable, and controllable. Through... and The system can clearly identify whether each large model judge is "precise and objective", "prefers long text", "prefers short text", or "randomly judges", providing a calculable and verifiable data foundation for subsequent dynamic weight adjustments, circuit breaker mechanisms, and security controls.

[0119] In some embodiments, the credibility assessment and dynamic weight adjustment of the agent include:

[0120] The discrimination parameters of each agent obtained from the optimal parameter solution set and preference parameters An assessment will be conducted, as follows:

[0121] When the intelligent agent Discrimination parameters When the value falls below a first preset threshold, a circuit breaker mechanism is triggered to shut down the intelligent agent. Remove from evaluation matrix A;

[0122] When the intelligent agent Preference parameters | When the threshold is exceeded, the agent's power level is reduced. Weighting in subsequent evaluation tasks.

[0123] In some embodiments, the first preset threshold refers to a pre-set lower limit threshold for the discrimination parameter, which is configured by the system according to the scene stability requirements, and is typically taken as the threshold for all agents after model convergence. The statistical quantile. A value below this first preset threshold indicates that the agent's judgment is nearly random and lacks effective evaluation capability.

[0124] The circuit breaker mechanism is a highly reliable security protection mechanism. When it is detected that the agent's discrimination parameter is too low or the judgment is completely random, it is immediately removed from the evaluation matrix permanently or temporarily to prevent it from polluting the overall evaluation results.

[0125] The second preset threshold refers to the threshold used to identify extreme style preferences. Exceeding the second preset threshold value indicates that the agent is severely bound by superficial features such as text length and information dispersion, and loses its ability to make objective evaluations.

[0126] Dynamic weight adjustment refers to the real-time and adaptive allocation of the agent's voice in evaluation and decision-making based on the agent's discriminative ability and the degree of extreme preferences, so as to achieve "weighting the capable and de-weighting the biased".

[0127] Credibility assessment refers to... and The AI's evaluation capabilities, objectivity, and stability are quantitatively scored to form a monitorable and auditable credibility system for AI judges.

[0128] After the model optimization is completed and the optimal parameter set is obtained, the system evaluates each agent in the evaluation matrix. Perform two rounds of quantitative judgment:

[0129] Round 1: Based on discrimination parameters The validity determination. This directly reflects the ability of an intelligent agent to distinguish between superior and inferior schemes. A higher score indicates stable, sharp, and reliable judgment. Too low a value indicates that the judgment is close to random guessing, contributing nothing to the evaluation or even introducing noise. When the value falls below the first preset threshold, the system determines that the agent is an invalid evaluation node, immediately triggers the circuit breaker mechanism, removes it from the current evaluation matrix, and prevents it from participating in any subsequent calculations and decisions.

[0130] Round Two: Based on Preference Parameters | The objectivity of the judgment. |Reflects the degree to which an intelligent agent is affected by surface features:| |A smaller value indicates neutrality and objectivity;| | "extremely" indicates an extreme preference for long texts or extremely concise answers, completely losing neutrality. When | When the deviation exceeds the second preset threshold, the system determines that the agent is an extreme bias node and dynamically reduces its weight according to the degree of deviation, thereby reducing its decision weight in subsequent evaluation tasks and weakening its impact on the overall result.

[0131] This mechanism completely abandons traditional static strategies such as "average voting" and "fixed weighting," and adopts dynamic trust routing based on parameter profiling. This ensures that the evaluation system always retains intelligent agents with high discriminative power, high objectivity, and low bias, and automatically filters out low-quality, highly biased, and randomized nodes.

[0132] This embodiment provides industrial-grade robustness and security for the evaluation system through a dynamic governance mechanism, with the following beneficial effects:

[0133] First, it fundamentally prevents the disruption of results caused by randomly assigned scores by agents. In multi-model collaborative evaluation, agents with weaker base models, unreasonable prompts, or misaligned parameters may assign random scores, which traditional methods cannot identify. This invention addresses this issue by... Threshold circuit breaking enables automatic removal of invalid nodes.

[0134] Second, it precisely suppresses systematic biases caused by extreme style preferences. Large model groups are prone to collectively favoring long texts and comprehensive descriptions, leading to erroneous but verbose solutions scoring higher. This invention addresses this by... Thresholds and dynamic weighting prevent extreme preferences from influencing the final outcome.

[0135] Third, it achieves closed-loop self-optimization of the evaluation system. As usage increases, the system continuously eliminates untrusted nodes and strengthens trusted nodes, resulting in continuous improvement in evaluation accuracy, stability, and objectivity. No manual recalibration is required, making it particularly suitable for systems with long-term operation and high reliability requirements, such as financial approval and policy Q&A.

[0136] Fourth, it meets the traceability requirements of heavily regulated scenarios. Every circuit breaker and every reduction in weight is handled by... and The quantitative numerical triggers are based on clear evidence, are recordable, reviewable, and auditable, and fully comply with the compliance requirements of government, medical, and financial systems.

[0137] In some embodiments, the differences in model architecture, parameter configuration, or evaluation strategy of the heterogeneous agents are achieved through at least one of the following methods:

[0138] In terms of model architecture, large language models with different network structures, parameter counts, or training datasets are used as the base models for each agent.

[0139] In terms of evaluation strategy, each agent is configured with prompt word templates that focus on different evaluation dimensions or have different logical structures;

[0140] In terms of parameter configuration, different sampling temperature parameters, top-p parameters, or output length limit parameters are set for each agent.

[0141] In this embodiment, the prompt word template refers to the instruction text used to guide the large model judges in performing the evaluation task. It directly determines the evaluation logic, focus dimensions, and judgment criteria, and is the key to constructing a heterogeneous evaluation strategy.

[0142] Sampling temperature refers to the parameter that controls the randomness of the output of a large model. The lower the temperature, the more deterministic and consistent the output; the higher the temperature, the more diverse and creative the output.

[0143] The top-p parameter refers to the kernel sampling parameter, which controls the range of probability concentration in the output text and affects the stability and diversity of the evaluation results.

[0144] Output length constraints refer to the restrictions on the output length of an agent's evaluation reasoning process, which indirectly affects its evaluation behavior and preference tendencies.

[0145] This embodiment clearly defines the specific construction method of heterogeneous intelligent agents from three levels: model, policy, and parameters, ensuring that the evaluation matrix has sufficient diversity and coverage, specifically manifested as follows:

[0146] (1) The heterogeneous implementation system at the model architecture level selects large language models with different network structures, parameter sizes, and pre-trained datasets as the evaluation base, such as open-source models with different parameter sizes, closed-source models trained by different vendors, and specialized models fine-tuned for different domains. Different base models have natural differences in knowledge structure, reasoning ability, and preference tendencies, which can avoid the common deviations brought about by homogeneous models.

[0147] (2) The heterogeneous implementation system at the evaluation strategy level designs differentiated prompt word templates for different agents, enabling each agent to focus on different evaluation dimensions: some agents focus on policy accuracy, some on compliance completeness, some on the rigor of expression, and some on process matching degree. Different evaluation logics enable agents to form an evaluation system with cross-validation and complementary coverage.

[0148] (3) The heterogeneous implementation system at the parameter configuration level sets different sampling temperature, top-p, output length limit and other generation parameters for each agent, so that the evaluation behavior presents differences in stability, diversity and consistency, and further improves the overall anti-interference ability and robustness.

[0149] The above three heterogeneous methods can be used individually or in any combination to ultimately form an evaluation matrix with multiple perspectives, complementary logic, and hierarchical capabilities.

[0150] Evaluation systems based on a single model, strategy, and parameter are highly susceptible to fixed biases and collective misjudgments. Heterogeneous intelligent agents achieve diversification at multiple levels—model, strategy, and parameter—making evaluation results closer to collective intelligence and significantly improving objectivity and reliability. In dynamic scenarios with frequent updates to financial policies and medical insurance rules, heterogeneous evaluation matrices can adapt to changes more quickly; the failure of a single model or preference drift will not lead to overall collapse, and the system possesses stronger fault tolerance and adaptability.

[0151] In some embodiments, in step S3, the parameterized model is constructed by modeling the specific feature dimension as a set of confusion variables to be controlled. The specific implementation steps are as follows:

[0152] Decouple the candidate solutions by features, and decouple each candidate solution Si Represented as the decoupled feature pair (X) i Z i ), where X i Z is the core quality feature vector, used to characterize the intrinsic attributes directly related to the accuracy and completeness in the preset evaluation dimensions; i To obfuscate the feature vector, which is used to characterize the latent semantic features extracted from the content of the candidate scheme that constitute the specific feature dimension;

[0153] Based on the observation dataset D k With the decoupled feature pair (X) i Z i A structural causal model is constructed, and the model parameters of the structural causal model are solved by minimizing the composite loss function to estimate the core quality feature vector X. i The average causal effect of the difference in the agent’s pairwise comparison preference judgments;

[0154] The composite loss function L causal The representation is as follows:

[0155] L causal =L fit +β·L balance ;

[0156] Among them, L fit This is a data fitting term used to correlate the probability predicted by the structural causal model with the indicator variable. Consistent; L balance This is a confusion balancing term used to weaken the confusion in the feature vector Z during the optimization process. i With core quality feature vector X i The statistical dependence between them is used to simulate randomized controlled trials and control the confusion feature vector {Z} of all candidate protocols. i The influence of latent semantic features represented by}, where β is a balancing hyperparameter;

[0157] In step S4, based on the optimal parameter solution set, the global preference score of each candidate scheme is calculated and sorted, including:

[0158] Based on the solved structural causal model, each candidate solution S i The confusion feature vector Z i The intervention is a pre-defined baseline vector Z0, while maintaining the candidate solution S. i Core quality feature vector X i Under the same counterfactual condition, calculate the candidate scheme S determined by all agents. i The average predicted probability that outperforms any other candidate solution is used as the candidate solution S. iThe final global preference score.

[0159] In some embodiments, feature decoupling refers to splitting the intrinsic quality and surface interference features of a candidate scheme into two sets of independent, non-redundant, and uncoupled feature representations in the mathematical vector space, so that the core quality and confusion features can be controlled, analyzed, and intervened in separately.

[0160] Core quality feature vector X i Core accuracy in the corresponding two-dimensional model It is the essential characteristic that determines the compliance, correctness, and policy fit of a plan, and it is the sole objective variable for evaluation.

[0161] Confusion feature vector Z i Information divergence in the corresponding two-dimensional model These include potential semantic features such as text length, information redundancy, level of detail in expression, writing style, and extended information, which are sources of systemic bias that need to be controlled and eliminated.

[0162] Structural causal models are used to model the evaluation process from the perspective of causal inference, distinguishing between the causal relationship of "core quality → evaluation result" and the spurious relationship of "confounding features → evaluation result", thus ensuring the objectivity and interpretability of the evaluation results from a theoretical perspective.

[0163] The average causal effect refers to the strength of the true and pure influence of core quality differences on an agent's preference judgment after excluding the interference of confusing features. It is the core basis for unbiased ranking.

[0164] Data fitting term L fit This refers to ensuring that the prediction preferences of the structural causal model are consistent with the actual comparison results in the observed dataset, thereby ensuring the model's accuracy in fitting the actual evaluation behavior.

[0165] Confusion balance term L balance This refers to the feature Z used to weaken confusion during model training. i With core quality feature vector X i The correlation between them makes them approach statistical independence, thus achieving the control effect of a randomized controlled trial (RCT).

[0166] Counterfactual intervention refers to the process of adding confusing features Z during the model inference stage. i Fixed as a unified reference vector Z0, only the core quality feature vector X is retained. i The differences are thus completely removed at the computational level from the influence of obfuscated features on the score.

[0167] The baseline vector Z0 represents a neutral, unbiased, and interference-free state of confusion features. It is usually the zero vector or the mean vector of all sample confusion features, and is used to achieve a unified and fair intervention baseline.

[0168] The final global preference score is an unbiased score calculated under counterfactual fairness conditions, determined solely by core quality, and can be directly used for ranking and outputting the optimal solution.

[0169] The working principle of the method involved in this embodiment is as follows:

[0170] First, the system performs strict feature decoupling on the candidate solutions, separating each candidate solution S... i Mapped to a pair of independent eigenvectors (X) i Z i ), where X i Focusing on defining the core correctness, compliance, and completeness of the solution, Z i It focuses on characterizing potential semantic confusion features such as text length, redundant information, and level of detail.

[0171] Subsequently, the system is based on the observation dataset D k With decoupling feature pair (X) i Z i Construct a structural causal model. The core objective of this structural causal model is to control for the confounding feature Z. i Under the condition of accurately estimating the core quality characteristic X i The true causal effect of the agent's preference judgments, rather than a simple fitting correlation.

[0172] To achieve this goal, the system is designed with a composite loss function: L causal =L fit +β·L balance , where L fit Ensure that model predictions are consistent with observed data; L balance By constraining Z i With X i To ensure statistical independence, a randomized controlled trial is simulated, eliminating spurious correlations caused by confounding features. By minimizing this loss, the model can learn a causal relationship structure unaffected by confounding features.

[0173] In the final scoring and ranking stage, the system performs counterfactual intervention reasoning, specifically as follows: It analyzes the confounding features Z of all candidate solutions. i Forced replacement with a unified baseline vector Z0, while fully preserving the core quality feature vector X. i Unchanged. Under this fair setting, calculate S for each candidate solution. i The average prediction probability that the agent determines is superior to other options is used as the final global preference score. This process completely eliminates length bias, style bias, and redundancy bias at the causal level, resulting in a completely pure, objective, and interpretable ranking of options.

[0174] The above approach eliminates spurious correlation bias at its theoretical root. Existing technologies merely fit the statistical relationship between evaluated behaviors and surface features, making them highly susceptible to being misled by confounding features and leading to erroneous conclusions. This invention identifies true causal relationships through a structural causal model, resulting in evaluation results that more closely resemble the objective judgment of human experts, thus possessing greater persuasiveness and credibility.

[0175] Compared to regularization constraints, counterfactual intervention completely eliminates the influence of confusing features on scoring by "giving a unified benchmark", which can achieve mathematically rigorous and unbiased evaluation results. It is particularly suitable for zero-fault-tolerance scenarios such as financial approval and medical insurance policies.

[0176] In some embodiments, calculating the global preference score for each candidate solution and ranking the output includes:

[0177] Based on the optimal parameter solution set, each agent The preference parameters for each agent Define an independent evaluation function F k (·);

[0178] For each candidate solution S i The candidate solution S i The content features are input into all K evaluation functions {F k} Calculate K scalar output values ​​respectively, and arrange these K scalar output values ​​in order to form scheme S. i K-dimensional utility vector V i The calculation formula is as follows:

[0179] V i =(F1(S i ), F2(S i ),...,F K (S i ), where the k-th component F k (S i S is a candidate solution. i The independent utility value under the evaluation function defined by agent k;

[0180] The K-dimensional utility vector {V} of all candidate solutions i} is mapped to a K-dimensional target space with K agents as orthogonal basis vectors, and each candidate solution S i This corresponds to a coordinate point in the space;

[0181] In the K-dimensional target space, the non-dominated solution set is selected according to the Pareto dominance criterion, specifically including: for a candidate solution S i If there is no other candidate solution S j This makes S j utility vector Vj It is greater than or equal to S in every dimension k. i utility vector V i The corresponding component of , and there exists at least one dimension k' such that ( > ( If ) is true, then S is determined to be true. i For a Pareto nondominated solution, the set of all Pareto nondominated solutions constitutes the Pareto optimal solution set.

[0182] Output the Pareto optimal solution set, and for each candidate solution S in the Pareto optimal solution set... i Perform the following steps to generate a report on its strengths:

[0183] Iterate through each evaluation dimension k and check if, within the Pareto optimal solution set, except for S... i Besides itself, is there another candidate solution S? j Its utility value F on that dimension k k (S j )>F k (S i );

[0184] There will be no such candidate solution S. j The indices of dimension k are collected and recorded as an index set, and the index set is used as a candidate scheme S. i The advantage dimension report is output together with the Pareto optimal solution set.

[0185] In some embodiments, the independent evaluation function F k (·) refers to the utility calculation function determined by the parameters of the k-th agent, used to output the candidate solution S. i The independent utility score under this agent evaluation criterion reflects the agent's specific evaluation of the solution.

[0186] K-dimensional utility vector V i This refers to combining the independent evaluation scores of K agents into a vector to fully characterize the comprehensive performance of the scheme from multiple dimensions and perspectives.

[0187] The K-dimensional target space refers to a vector space constructed with each agent as an independent coordinate axis, used to intuitively present the distribution of the advantages and disadvantages of each scheme under multi-agent evaluation.

[0188] The Pareto dominance criterion is a core criterion for determining the best or worst outcome in multi-objective optimization. It is used to select the globally optimal solution set when it is impossible to maximize all objectives simultaneously.

[0189] A Pareto nondominated solution is a solution for which there is no other solution that is not worse in all evaluation dimensions and is better in at least one dimension. It represents the optimal choice from a multi-objective perspective.

[0190] The Pareto optimal solution set refers to the set of all Pareto non-dominated solutions, which is the fairest and most uncontroversial set of candidate solutions under multi-agent evaluation.

[0191] The Advantages Dimension Report records which evaluation dimensions each solution possesses unique and insurmountable advantages, providing intuitive support for user selection, result interpretation, and decision-making.

[0192] This embodiment presents a multi-objective Pareto optimal ranking scheme under multi-agent evaluation, which is suitable for complex financial / policy question-answering scenarios that require simultaneous satisfaction of multiple evaluation perspectives and cannot be simply ranked by a single objective. The specific working principle is as follows:

[0193] First, the system constructs an independent evaluation function F for each agent based on the optimal parameters. k (·), this function has been stripped of bias and retains core quality, and can output an unbiased utility score.

[0194] Subsequently, for each candidate solution S i Calculate the K-dimensional utility vector V i =(F1(S i ), F2(S i ),...,F K (S i This vector maps the proposed solution to a K-dimensional target space. It fully preserves the independent evaluation information of each agent, without simple averaging or weighted fusion.

[0195] In K-dimensional space, the system selects non-dominated solutions based on the Pareto dominance criterion: if candidate solutions S i Not compared to any other candidate solution S j If S has a comprehensive advantage, then i This step yields the Pareto optimal solution. It preserves all fair and reasonable optimal solutions in the multi-perspective evaluation.

[0196] Finally, the system generates an advantage dimension report for each Pareto optimal solution: it iterates through all evaluation dimensions, marking which dimensions the solution is optimal within the solution set, with no other solution surpassing it. This report is output along with the optimal solution set, allowing users to clearly understand the advantages of each solution.

[0197] By employing the above methods, information loss caused by a single total score is avoided, providing a fairer, more transparent, and more interpretable evaluation result. In multi-agent heterogeneous evaluation scenarios, different evaluation perspectives are not forcibly unified, and no effective information from any dimension is lost, making the evaluation result more comprehensive and persuasive.

[0198] In complex scenarios such as financial policies and healthcare Q&A, different intelligent agents represent different compliance review dimensions. Pareto optimal solutions ensure that no solution is outperformed in all dimensions, preventing high-quality solutions from being incorrectly excluded due to weaknesses in a single dimension. By providing highly interpretable reports on strengths in various dimensions, users can intuitively see the unique advantages of each solution in terms of accuracy, completeness, and compliance, significantly improving user trust and decision-making efficiency, and meeting the transparency requirements of government, healthcare, and financial systems.

[0199] In some embodiments, a model robustness enhancement stage is included before step S3, the model robustness enhancement stage including the following steps:

[0200] Construct and train an adversarial sample generator, the adversarial sample generator using any candidate scheme S from the candidate scheme set S. i The original content is the input, and the output is an adversarial candidate solution S. advi The training objective of the generator is as follows: to make S advi With S i Maintain consistency across the core quality feature vectors defined by the parameterized model, while maximizing S. advi Feature strength in the latent semantic feature dimension;

[0201] Using the trained adversarial sample generator, an adversarial variant is generated for each scheme in the candidate scheme set S. All adversarial variants are then added to the candidate scheme set S to form an augmented candidate scheme set S. aug Based on S aug Then repeat step S2 to construct an augmented observation dataset containing the results of adversarial sample comparisons;

[0202] Step S3 is performed using the augmented observation dataset, and the loss function further includes an adversarial regularization term L. adv The L adv It is constructed as follows: for each pair of original candidate schemes S i Its corresponding adversarial candidate solution S advi The parameterized model is calculated as S. advi With S i Find the norm of the difference between the assigned core quality scores and minimize the sum.

[0203] In some embodiments, an adversarial sample generator refers to a neural network model specifically designed to generate interfering samples to simulate adversarial interference such as length attacks, redundancy attacks, and style attacks that may be encountered in large model evaluation.

[0204] Adversarial candidate solution S adviThis refers to a disruptive scheme that deliberately enhances obfuscated features (such as length, redundancy, and level of detail) while maintaining the core quality consistent with the original scheme. This is used to test and improve the model's resistance to bias.

[0205] Core quality consistency means that the adversarial sample and the original scheme are completely identical in terms of policy correctness, compliance, and factual accuracy, with differences only in superficial characteristics.

[0206] The strength of latent semantic features refers to the numerical value of the confusing features (length, redundancy, divergence). Maximizing this strength means constructing extreme biased interference samples.

[0207] The augmented set of candidate solutions S aug An expanded set of original schemes and adversarial examples is included to improve the model's generalization and anti-interference capabilities.

[0208] The augmented observation dataset contains pairwise comparisons of adversarial examples, which are used to enhance the model's immunity to obfuscated features during training.

[0209] Adversarial regularization term L adv是指加 The constraint term in the loss function forces the model to give consistent scores to candidate solutions with "same core quality but different surface features", thereby completely eliminating adversarial interference such as length bias and redundancy bias.

[0210] The method described in this embodiment can maintain stable and reliable evaluation even under extreme bias interference. The specific working principle is as follows:

[0211] First, the system constructs an adversarial sample generator, whose training objective is:

[0212] Maintaining adversarial candidate solutions S advi Compared with the original candidate solution S i The core quality is completely consistent;

[0213] To maximize the strength of confounding features (length, redundancy, divergence), construct extreme biased samples.

[0214] Subsequently, the generator generates adversarial variants for each original scheme, which are added to the candidate set to form an augmented candidate scheme set S. aug And based on S aug Re-execute pairwise comparisons to construct an augmented observation dataset containing adversarial examples.

[0215] During the model training phase, an adversarial regularization term L is added to the original loss function. adv The calculation formula is as follows:

[0216] L adv =Σ‖Score(S advi )-Score(Si )‖;

[0217] Here, Score(·) represents the core quality score calculation function, and ||·|| represents the norm calculation. An adversarial regularization term forces the model to output consistent scores for samples with "same core but different surface appearances," thus learning during training to completely ignore confounding features and focus solely on core quality. Through adversarial training, the model acquires extremely strong resistance to interference and bias, enabling it to provide stable, objective, and unbiased evaluation results even when faced with deliberately constructed, lengthy, redundant, or illusionary text.

[0218] In scenarios such as finance and government affairs, there may be deliberately lengthy, over-packaged, or falsely comprehensive misleading answers. This invention, through adversarial training, enables the model to completely "ignore" such interference, relying solely on core quality scores, which significantly improves the model's stability and generalization ability in extreme scenarios. The adversarial regularization term forces the model to give the same score for the same core quality, further consolidating the effects of feature separation and bias removal, and improving the stability and reliability of the evaluation system.

[0219] In some embodiments, the credibility assessment and dynamic weight adjustment of the agent further includes:

[0220] Based on the trained parameterized evaluation model, the calculation of each agent... The preference feature vector is calculated, and the cosine similarity between the preference feature vector and the consensus preference vector of all agents is calculated to obtain the consensus deviation δ. k Simultaneously, computational intelligent agents In the observation dataset D k Internal consistency score κ of the judgment results k If the intelligent agent Simultaneously satisfying δ k >τ1 and κ k >τ2, then the agent Identify as a highly consistent dissenter, where τ1 is a preset first score threshold and τ2 is a preset second score threshold;

[0221] Cluster analysis was performed on all preference feature vectors identified as having high consistency and dissent to identify a set of common preference patterns that significantly deviate from the consensus {P1, P2, ..., P...}. m}, where m is the total number of preference patterns;

[0222] For each identified preference pattern P tThe value of t ranges from [1, m]. Perform the following operations: construct a positive sample set from the winning candidate solutions preferred by all dissenting agents belonging to this preference pattern, and construct a negative sample set from their corresponding losing solutions; extract a set of features that can best explain the preference pattern P from the original content features of the positive and negative sample sets using feature importance analysis or gradient inversion methods. t Key content feature set {F t};

[0223] Key content features F t Input a pre-trained natural language generation model, and drive the pre-trained natural language generation model to synthesize a segment describing the key content features F. t The natural language definition text of the potential evaluation perspective or principle represented, and the natural language definition text is used as an evaluation dimension to expand the proposal output.

[0224] In this embodiment, the consensus deviation δ k This refers to the degree of difference between the consensus of an individual agent and that of the group, δ k The larger the value, the more unique and unconventional the agent's evaluation tendencies are.

[0225] Internal consistency score κ k The self-consistency, stability, and consistency of the agent's self-evaluation behavior are used to assess the agent's self-evaluation behavior. k The higher the value, the more stable and consistent the agent's judgment logic.

[0226] Highly consistent dissenters are agents who deviate significantly from the group but whose own judgments are highly stable and consistent. They may represent undiscovered potential evaluation dimensions or reasonable professional perspectives.

[0227] Preference pattern clustering refers to grouping dissenting preference vectors, identifying preference types with common deviation characteristics, and extracting systematic dissenting patterns.

[0228] Key content feature set F t是指驱 The core textual features of a certain preference pattern, such as "emphasizing process", "focusing on evidence", and "preferring legal citations", are professional evaluation dimensions.

[0229] The proposal to expand the evaluation dimensions refers to transforming implicit preference patterns into explicit, understandable, and reusable evaluation rules, which are used to expand, improve, and upgrade the existing evaluation system.

[0230] This embodiment provides a high-level intelligent mechanism that moves from "passively eliminating bias" to "actively discovering evaluation dimensions," enabling the system not only to eliminate bias but also to autonomously evolve its evaluation capabilities. The specific working principle is as follows:

[0231] First, the system calculates the consensus deviation δ for each agent.k With internal consistency score κ k The process involves filtering out dissenting voices with high consistency: these agents, though deviating from the mainstream, possess highly stable judgments and often represent overlooked professional perspectives.

[0232] Subsequently, the dissenter preference vectors were clustered to extract several common deviation patterns P. t We discovered assessment dimensions that were not captured by the group.

[0233] For each preference pattern, the system uses preferred options as positive samples and disliked options as negative samples. Through feature importance analysis or gradient inversion, it extracts the key content features F that drive the preference. t For example, whether it includes reimbursement vouchers, whether it cites the latest policy document number, and whether it emphasizes the approval process.

[0234] Finally, the key content features F t The system inputs a natural language generation model to synthesize human-understandable descriptive text for assessment dimensions, generating proposals to expand these dimensions. These proposals are then used to upgrade prompts, refine assessment rules, and expand policy review points. This mechanism transforms the assessment system from a "passive correction" approach to an "active evolution" one, continuously and autonomously discovering more professional, comprehensive, and business-relevant assessment dimensions.

[0235] The aforementioned solution transforms dissenting opinions from distractions into valuable sources of professional perspectives. Traditional systems treat dissenters as noise; this invention identifies high-value dissenting opinions and uncovers reasonable review dimensions overlooked by the mainstream. In scenarios where financial policies and medical insurance rules are constantly being updated, the system can autonomously discover new assessment points, compliance dimensions, and review priorities without requiring manual redesign of prompts or rules, significantly reducing system maintenance costs. Ultimately, it outputs understandable, reusable, and implementable proposals for expanding assessment dimensions, transforming implicit patterns discovered by AI into explicit rules that can be executed by humans. These rules can be directly used to optimize manual review processes, upgrade policy knowledge bases, and improve financial approval standards.

[0236] In practical applications, this invention primarily targets intelligent financial approval and open-ended Q&A scenarios related to financial / medical insurance policies in public institutions such as hospitals and public service units. The natural language queries submitted by users typically involve highly specialized and compliant content such as tax deduction standards for outpatient consultation fees from external experts, types of supporting documentation required for reimbursement, financial approval workflow nodes, and policy applicability conditions. When generating a set of candidate answers, the system can output from different generation subsystems or different large models. Different candidate solutions exhibit significant differences in core policy statements, information completeness, and the amount of extended content. Some solutions only provide the core conclusions, while others include extended content such as tax exemption limits, contract templates, and approval processes, thus forming a differentiated pool of candidate answers.

[0237] The evaluation system adopted in this invention abandons the simple average weighting, direct voting, and fixed weighting mechanisms commonly used in traditional multi-model evaluations. These traditional mechanisms assume all evaluation models have the same judgment ability, ignoring the differences in capabilities among different models in professional fields such as financial policy and medical insurance clauses. This can easily lead to the evaluation results being dragged down by weaker models. Furthermore, traditional large-model judging methods often suffer from circular bias problems when making pairwise comparisons. This means that there may be logical inconsistencies where a model judges candidate solution A as superior to candidate solution B, candidate solution B as superior to candidate solution C, and candidate solution C as superior to candidate solution A. This renders the ranking results based on comparison probabilities unreliable and unconvincing.

[0238] To address the aforementioned issues, this invention employs an unsupervised joint optimization architecture. Without relying on financial expert annotations or introducing human benchmark answers, it utilizes self-supervised feedback signals generated through pairwise comparisons among multiple agents to simultaneously determine the quality of candidate solutions and the reliability of the agents. During model optimization, the system uses the Alternating Directional Multiplier Method (ADMM) or gradient descent algorithm to solve the loss function, ensuring parameter convergence stability and reproducible numerical results, thereby maintaining evaluation consistency even in dynamically updated policy scenarios.

[0239] After completing parameter solving and bias removal, this invention not only ranks candidate solutions and outputs the optimal solution based on core accuracy, but also generates a quantifiable and monitorable personality profile for each agent. This profile, composed of discrimination and preference parameters, clearly distinguishes different evaluation behavior patterns, such as high-discrimination-focused accuracy, low-discrimination-focused random judgment, preference for long texts, and preference for concise answers. Based on this personality profile, the system constructs a visualized AI reliability monitoring panel, enabling managers to intuitively grasp the working status, objectivity, and deviation level of each agent, meeting the stringent regulatory requirements of public financial systems for the explainability, controllability, auditability, and traceability of AI applications.

[0240] Furthermore, this invention, through preference-neutral regularization constraints, forcibly suppresses the collective preference of intelligent agents for superficial features such as text length, information redundancy, and content detail at the group level, thereby achieving mathematical stripping away of length bias. This ensures that the final output retains only core quality information such as policy accuracy and compliance, preventing candidate solutions from receiving higher scores due to lengthy content, comprehensive descriptions, factual errors, or policy illusions. This significantly improves the reliability and security of open-ended financial policy Q&A.

[0241] In a second aspect, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the intelligent evaluation method based on bias elimination and feature separation as described in the first aspect of the present invention.

[0242] The computer-readable storage medium may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory.

[0243] The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD ROM); the magnetic surface memory may be a disk storage device or a magnetic tape storage device.

[0244] The volatile memory may be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synclink dynamic random access memory (SLDRAM), and direct memory bus random access memory (DRRAM). The computer-readable storage media described in the embodiments of the present invention are intended to include these and any other suitable types of memory.

[0245] like Figure 2 As shown, in a third aspect, the present invention provides an electronic device 10, including a processor 101 and a storage medium 102, wherein a computer program is stored on the storage medium, and the computer program, when executed by the processor, implements the intelligent evaluation method based on bias elimination and feature separation as described in the first aspect of the present invention.

[0246] In some embodiments, the processor may be implemented by software, hardware, firmware, or a combination thereof, and may use at least one of the following: circuit, single or multiple application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), central processing units (CPUs), controllers, microcontrollers, and microprocessors, thereby enabling the processor to execute some or all of the steps or any combination thereof in the intelligent evaluation method based on bias elimination and feature separation described in the various embodiments of the present invention.

[0247] Finally, it should be noted that although the above embodiments have been described in the description and drawings of this invention, this should not limit the scope of patent protection of this invention. Any technical solutions that are based on the essential concept of this invention, utilize the content described in the description and drawings of this invention to make equivalent structural or procedural substitutions or modifications, as well as the direct or indirect application of the technical solutions of the above embodiments to other related technical fields, are all included within the scope of patent protection of this invention.

Claims

1. An intelligent evaluation method based on bias elimination and feature separation, characterized in that, The method includes: S1: Receive a natural language query input by the user, and generate a set of M candidate solutions based on the natural language query. And schedule K heterogeneous agents to form an evaluation matrix. The heterogeneous intelligent agents differ in model architecture, parameter configuration, or evaluation strategy. S2: For each agent, schedule the agent to perform pairwise comparisons of candidate solutions in the candidate solution set S. For any two candidate solutions S i and S j By the kth intelligent agent Determine and generate indicator variables When the intelligent agent Determine candidate solution S i It outperforms candidate solution S in the preset evaluation dimensions. j hour, =1, otherwise =0, the preset evaluation dimensions include the accuracy, completeness, and relevance to the natural language query of the candidate solutions, and all comparison results are collected to construct an observation dataset. ={(S i ,S j , )|S i ,S j ∈S, i≠j}; S3: Based on observation dataset A parameterized model is constructed to characterize the preference features of each agent, and the optimal parameter solution set of the parameterized model is solved by minimizing the loss function containing preference constraints. The preference constraints are used to constrain the overall preference strength of the model parameters for a specific feature dimension in order to eliminate systematic bias in the evaluation process. The specific feature dimension is a latent semantic feature dimension obtained by learning the representation of the content of the candidate scheme. S4: Based on the optimal parameter solution set, calculate the global preference score of each candidate scheme and sort and output it. According to the degree of deviation between the preference feature vector of each agent and the consensus preference vector reflected in the optimal parameter solution set, evaluate the credibility of the agent and adjust the dynamic weight. In step S3, constructing a parameterized model to characterize the preference features of each agent includes: Each candidate solution is vectorized into intrinsic attributes of two dimensions: core accuracy and information divergence. Here, candidate solution S... i Core accuracy is denoted as Candidate Solution S i Information divergence is denoted as ; For each intelligent agent Configure discrimination parameters and preference parameters ; Building intelligent agents For candidate solution S i Comprehensive perceived utility function The formula is as follows: ； intelligent agent Determine candidate solution S i It outperforms candidate solution S in the preset evaluation dimensions. j probability distribution function as follows: ； The loss function that includes preference constraints is denoted as The calculation formula is as follows: ； Where K represents the total number of agents, and k represents the index of the agent. This represents a hyperparameter used to control the weights of preference constraint terms. This represents a preference constraint term, used to force the overall preference of the agent group for information divergence characteristics to tend towards neutrality.

2. The intelligent evaluation method based on bias elimination and feature separation as described in claim 1, characterized in that, The credibility assessment and dynamic weight adjustment of the agent include: The discrimination parameters of each agent obtained from the optimal parameter solution set and preference parameters An assessment will be conducted, as follows: When the intelligent agent Discrimination parameters When the value falls below a first preset threshold, a circuit breaker mechanism is triggered to shut down the intelligent agent. Remove from evaluation matrix A; When the intelligent agent Preference parameters | When the threshold is exceeded, the agent's power level is reduced. Weighting in subsequent evaluation tasks.

3. The intelligent evaluation method based on bias elimination and feature separation as described in claim 1, characterized in that, The heterogeneous intelligent agents exhibit differences in model architecture, parameter configuration, or evaluation strategy through at least one of the following methods: In terms of model architecture, large language models with different network structures, parameter counts, or training datasets are used as the base models for each agent. In terms of evaluation strategy, each agent is configured with prompt word templates that focus on different evaluation dimensions or have different logical structures; In terms of parameter configuration, different sampling temperature parameters, top-p parameters, or output length limit parameters are set for each agent. The top-p parameter refers to the kernel sampling parameter, which is used to control the probability set range of the output text.

4. The intelligent evaluation method based on bias elimination and feature separation as described in claim 1, characterized in that, The calculation and ranking of global preference scores for each candidate solution includes the following output: Based on the optimal parameter solution set, each agent The preference parameters for each agent Define an independent evaluation function F k (·); For each candidate solution S i The candidate solution S i The content features are input into all K evaluation functions {F k } Calculate K scalar output values ​​respectively, and arrange these K scalar output values ​​in order to form scheme S. i K-dimensional utility vector V i The calculation formula is as follows: V i =(F1(S i ), F2(S i ),...,F K (S i ), where the k-th component F k (S i S is a candidate solution. i The independent utility value under the evaluation function defined by agent k; The K-dimensional utility vector {V} of all candidate solutions i } is mapped to a K-dimensional target space with K agents as orthogonal basis vectors, and each candidate solution S i This corresponds to a coordinate point in the space; In the K-dimensional target space, the non-dominated solution set is selected according to the Pareto dominance criterion, specifically including: for a candidate solution S i If there is no other candidate solution S j This makes S j utility vector V j It is greater than or equal to S in every dimension k. i utility vector V i The corresponding component of , and there exists at least one dimension k' such that ( > ( If ) is true, then S is determined to be true. i For a Pareto nondominated solution, the set of all Pareto nondominated solutions constitutes the Pareto optimal solution set. Output the Pareto optimal solution set, and for each candidate solution S in the Pareto optimal solution set... i Perform the following steps to generate a report on its strengths: Iterate through each evaluation dimension k and check if, within the Pareto optimal solution set, except for S... i Besides itself, is there another candidate solution S? j Its utility value F on that dimension k k (S j )>F k (S i ); There will be no such candidate solution S. j The indices of dimension k are collected and recorded as an index set, and the index set is used as a candidate scheme S. i The advantage dimension report is output together with the Pareto optimal solution set.

5. The intelligent evaluation method based on bias elimination and feature separation as described in claim 1, characterized in that, Prior to step S3, a model robustness enhancement stage is included, which comprises the following steps: Construct and train an adversarial sample generator, the adversarial sample generator using any candidate scheme S from the candidate scheme set S. i The original content is the input, and the output is an adversarial candidate solution S. advi The training objective of the generator is as follows: to make S advi With S i Maintain consistency across the core quality feature vectors defined by the parameterized model, while maximizing S. advi Feature strength in the latent semantic feature dimension; Using the trained adversarial sample generator, an adversarial variant is generated for each scheme in the candidate scheme set S. All adversarial variants are then added to the candidate scheme set S to form an augmented candidate scheme set S. aug Based on S aug Then repeat step S2 to construct an augmented observation dataset containing the results of adversarial sample comparisons; Step S3 is performed using the augmented observation dataset, and the loss function further includes an adversarial regularization term L. adv The L adv It is constructed as follows: for each pair of original candidate schemes S i Its corresponding adversarial candidate solution S advi The parameterized model is calculated as S. advi With S i Find the norm of the difference between the assigned core quality scores and minimize the sum.

6. The intelligent evaluation method based on bias elimination and feature separation as described in claim 1, characterized in that, The credibility assessment and dynamic weight adjustment of the agent also include: Based on the trained parameterized evaluation model, the calculation of each agent... The preference feature vector is calculated, and the cosine similarity between the preference feature vector and the consensus preference vector of all agents is calculated to obtain the consensus deviation δ. k Simultaneously, computational intelligent agents In the observation dataset D k Internal consistency score κ of the judgment results k If the intelligent agent Simultaneously satisfying δ k >τ1 and κ k >τ2, then the agent Identify as a highly consistent dissenter, where τ1 is a preset first score threshold and τ2 is a preset second score threshold; Cluster analysis was performed on all preference feature vectors identified as having high consistency and dissent to identify a set of common preference patterns that significantly deviate from the consensus {P1, P2, ..., P...}. m }, where m is the total number of preference patterns; For each identified preference pattern P t The value of t ranges from [1, m]. Perform the following operations: construct a positive sample set from the winning candidate solutions preferred by all dissenting agents belonging to this preference pattern, and construct a negative sample set from their corresponding losing solutions; extract a set of features that can best explain the preference pattern P from the original content features of the positive and negative sample sets using feature importance analysis or gradient inversion methods. t Key content feature set {F t }; Key content features F t Input a pre-trained natural language generation model, and drive the pre-trained natural language generation model to synthesize a segment describing the key content features F. t The natural language definition text of the potential evaluation perspective or principle represented, and the natural language definition text is used as an evaluation dimension to expand the proposal output.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the intelligent evaluation method based on bias elimination and feature separation as described in any one of claims 1 to 6.

8. An electronic device having a computer program stored thereon, characterized in that, It includes a processor and a storage medium, wherein a computer program is stored on the storage medium, and the computer program, when executed by the processor, implements the intelligent evaluation method based on bias elimination and feature separation as described in any one of claims 1 to 6.

Images