Human-machine cooperation method for solving human deviation

Abstract

The invention discloses a man-machine cooperation method for solving human deviation. The man-machine cooperation method comprises the steps that initialization is carried out; iteratively executing batch Thompson sampling, batch Thompson sampling, preference query and data updating and Gaussian process posteriori updating until the maximum number of iterations is reached; and after iteration is finished, returning an optimal action corresponding to the maximum potential function value in the action space. According to the embodiment, the long-tail preference relationship problem is fundamentally solved. In the batch Thompson sampling stage, diversified candidate preference pairs are generated through an adaptive covariance scaling factor and a double-independent sampling mechanism; in the sub-mode marginal gain evaluation stage, the dominant effect of head preference is effectively inhibited by utilizing the profit decreasing characteristic and marginal gain maximization of a sub-mode function; and in a Gaussian process posteriori updating stage, the observed preferences are integrated into a Bayesian framework, and posteriori distribution is refined to guide subsequent sampling. By optimizing the preference learning process, the preference query times are remarkably reduced, and the learning efficiency and accuracy are improved.

Description

Technical Field

[0001] This invention relates to the field of human-computer collaboration technology, and in particular to a human-computer collaboration method for addressing human biases. Background Technology

[0002] In recent years, learning from human preferences has become a key research topic in artificial intelligence, providing a promising approach to aligning autonomous machines with human goals. Preference learning enables a range of human-machine collaborative applications, including autonomous driving, medical diagnosis, natural language processing, and physical rehabilitation. By incorporating human feedback into the learning loop, AI systems achieve stronger generalization capabilities, more reliable reasoning, and improved context awareness, enhancing their ability to collaborate effectively with humans.

[0003] Despite their wide applicability, human preference data is inherently scarce and costly to acquire. Directly modeling diverse human preferences using mathematical tools remains a nontrivial challenge. A prominent technical direction involves reinforcement learning from human feedback or preferences, such as Reinforcement Learning from Human Feedback (RLHF), Direct Preference Optimization (DPO), and RL from Diverse Preferences. These methods aim to adapt RL policies to external signals by constructing reward functions or reward models from large-scale human preference data. Another complementary research path studies preference learning through Bayesian optimization, focusing on building interactive decision-making systems capable of efficiently querying and integrating human preferences.

[0004] A common assumption in previous work is that human preferences can be represented as pairwise comparisons between instances or actions. The learning objective then becomes how to infer underlying preference rankings from these pairwise preference relationships. For example, a human might have preferences for two given actions. and Expressing preferences, among which express Superior This is detailed in the Bradley-Terry (BT) model. Recent analyses, such as those by Mohammad et al., have highlighted the limitations of pairwise preference-dependent RL algorithms, particularly regarding the reference policy. The applied weak regularization. However, a broader understanding of how preference bias affects the performance of the learning paradigm remains limited.

[0005] Existing Pairwise Gaussian Process (PGP) methods learn human preferences from pairwise comparisons by imposing a Gaussian process prior on the latent utility function and then using a sigmoid likelihood function to model the pairwise comparisons, thereby inferring the latent preference ranking. However, existing methods suffer from an important but under-recognized problem: the long-tailed preference phenomenon. Traditional PGP methods are prone to getting suboptimal under long-tailed preference relationships.

[0006] When a certain action is repeatedly marked as optimal, a dominance effect occurs: this action dominates subsequent pairwise comparisons, suppressing tail preference signals and naturally suppressing rare or uncommon preferences, thus forming a long-tailed preference distribution. The mathematical root of this dominance effect lies in the saturation property of the sigmoid likelihood function: once the current optimal action... Continue to win comparisons, score difference ,in and Representing candidate solutions respectively and The potential function value, This is the noise tolerance parameter. and The index for candidate solutions becomes highly skewed. ), sigmoid function It tends to saturate, causing its derivative to... This leads to paired interactive items The likelihood of the new tail action tends to zero. This means that the likelihood update of the newly proposed tail action is actually negligible, while the regularization effect accumulated on the best action continues to grow, eventually preventing the new action from affecting the model and causing the algorithm to get stuck in a suboptimal local solution.

[0007] Therefore, it is necessary to improve one or more of the problems existing in the above-mentioned related technical solutions.

[0008] It should be noted that this section is intended to provide background or context for the technical solutions of this disclosure as set forth in the claims. The description herein does not constitute an admission that it is prior art simply because it is included in this section. Summary of the Invention

[0009] The purpose of this invention is to provide a human-computer collaboration method for addressing human biases, thereby overcoming, at least to some extent, one or more problems caused by limitations and defects in related technologies.

[0010] This invention provides a human-computer collaboration method for addressing human biases, comprising: S1, Initialization: Set the action space, Gaussian process prior covariance, noise parameters, maximum number of iterations and batch size; initialize the preference set, access set, latent function value dictionary and Gaussian process posterior. S2, iteratively execute batch Thompson sampling, batch Thompson sampling, preference query and data update, and Gaussian process posterior update until the maximum number of iterations is reached: S21, Batch Thompson Sampling: Calculate the adaptive covariance scaling factor, and generate a set of candidate preference pairs based on the Gaussian process posterior of the previous iteration, the batch size, and the covariance scaling factor using a dual independent sampling mechanism. S22, Submodular Marginal Gain Evaluation: Based on the candidate preference pair set, access set, and latent function value dictionary, the information value of each candidate preference pair is quantified through the submodular evaluation function, the marginal gain of the candidate preference pair relative to the selected set is calculated, and the candidate preference pair that maximizes the marginal gain is selected to form the optimal subset; S23, Preference Query and Data Update: Perform human preference query on each preference pair in the optimal subset to obtain preference relationships and update the preference set, access set and latent function value dictionary; S24, Gaussian process posterior update: Based on the updated preference set, a regularized Bayesian update method is used to integrate observed preferences to refine the Gaussian process posterior, resulting in a posterior model containing the posterior mean, covariance eigenvector, covariance eigenvalue, and optimal action index. S3, after the iteration is complete, returns the optimal action corresponding to the maximum potential function value in the action space.

[0011] In this invention, in S21, the dual independent sampling mechanism of batch Thompson sampling is specifically as follows: two independent samplings are performed from the posterior model to obtain the latent function value, and the action with the largest latent function value in the two samplings is selected to form a candidate preference pair. This process is repeated until K candidate preference pairs are generated.

[0012] In this invention, the covariance scaling factor in S21 The expression is as follows:

[0013] in, It is the initial scaling factor. It is the attenuation rate. Indicates the iterative index.

[0014] In this invention, the modulus evaluation function in S22 G The expression is as follows:

[0015] in, For two actions in a candidate preference pair For action The corresponding latent function value, For access set.

[0016] In this invention, the preference pair is relative to the selected set. The marginal gain is:

[0017] in, For the currently selected preference pair set, For candidate preference pairs, For set The sum of the submodular evaluation function values ​​of all preference pairs.

[0018] In this invention, the submode marginal gain evaluation further includes a continuous pair detection step: Identify preference pair chains with transitive relational structures, and adjust the submodular evaluation score of the first preference pair in the chain to medium priority to balance exploration efficiency and information gain.

[0019] In this invention, in S24, during the Gaussian process posterior update, the posterior mean is obtained by minimizing the objective function. Defined as:

[0020] Among them, fractional difference , and Representing candidate solutions respectively and The potential function value, The covariance matrix of the Gaussian process is... For the log-likelihood term based on the submodular function, is the vector of potential functions.

[0021] The technical solution provided by this invention may include the following beneficial effects: This invention presents a human-computer collaboration method to address human bias, fundamentally resolving the problem of long-tailed preference relationships. In the batch Thompson sampling stage, an adaptive covariance scaling factor and a dual-independent sampling mechanism are used to conduct extensive exploration in the early stages and gradually focus on development in the later stages, generating diverse candidate preference pairs. In the submodulus marginal gain evaluation stage, the diminishing returns property of the submodulus function is utilized to quantify the information value of each preference pair. By maximizing the marginal gain, the most informative preference pair is selected, effectively suppressing the dominant effect of head preferences. In the Gaussian process posterior update stage, the observed preferences are integrated into a Bayesian framework, refining the posterior distribution to guide subsequent sampling. This framework significantly reduces the number of preference queries required by optimizing the preference learning process, improving learning efficiency and accuracy, and providing more reliable preference learning support for human-computer collaboration systems. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0023] Figure 1 This is a flowchart of the human-computer collaboration method for addressing human biases in this invention; Figure 2 This is a schematic diagram of the SubGP (Submodular Gaussian Process Preference Learning) framework described in this invention. Figure 3 A schematic diagram illustrating the long-tailed preference relationship phenomenon: a comparison of differences in arm distribution and convergence; Figure 4 This figure shows the performance comparison results of SubGP and traditional Pairwise GP in a constrained linear bandit environment in this invention. Detailed Implementation

[0024] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0025] Furthermore, the accompanying drawings are merely illustrative diagrams of embodiments of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0026] First, please refer to Figure 2 , Figure 2 A schematic diagram of the long-tailed preference relationship phenomenon: a comparison of differences in arm distribution and convergence. Figure 2The algorithm comprises four subplots. The first three subplots show the constraint value distribution under different bias intensity levels (Weak / Medium / Strong). The proportions of arms close to the optimal region (constraint value > 0.9) are 23%, 42%, and 58%, respectively, verifying that the density of high-value arms significantly increases with increasing bias intensity. The fourth subplot shows the distance bias convergence curves under the three bias intensities: under weak bias, the algorithm can converge to a near-optimal solution, but under strong bias, the distance bias remains above 15 and fluctuates wildly, making it impossible to distinguish the true optimal solution from nearby high-quality options. This systematically demonstrates that bias severely weakens the performance of the traditional PairwiseGP method.

[0027] Secondly, this example implementation provides a human-computer collaboration method for addressing human biases; please refer to [reference needed]. Figure 1 and Figure 2 This method may include: S1-S3, as follows: S1, Initialization: Set the action space, Gaussian process prior covariance, noise parameters, maximum number of iterations and batch size; initialize the preference set, access set, latent function value dictionary and Gaussian process posterior. S2, iteratively execute batch Thompson sampling, batch Thompson sampling, preference query and data update, and Gaussian process posterior update until the maximum number of iterations is reached: S21, Batch Thompson Sampling: Calculate the adaptive covariance scaling factor, and generate a set of candidate preference pairs based on the Gaussian process posterior of the previous iteration, the batch size, and the covariance scaling factor using a dual independent sampling mechanism. S22, Submodular Marginal Gain Evaluation: Based on the candidate preference pair set, access set, and latent function value dictionary, the information value of each candidate preference pair is quantified through the submodular evaluation function, the marginal gain of the candidate preference pair relative to the selected set is calculated, and the candidate preference pair that maximizes the marginal gain is selected to form the optimal subset; S23, Preference Query and Data Update: Perform human preference query on each preference pair in the optimal subset to obtain preference relationships and update the preference set, access set and latent function value dictionary; S24, Gaussian process posterior update: Based on the updated preference set, a regularized Bayesian update method is used to integrate observed preferences to refine the Gaussian process posterior, resulting in a posterior model containing the posterior mean, covariance eigenvector, covariance eigenvalue, and optimal action index. S3, after the iteration is complete, returns the optimal action corresponding to the maximum potential function value in the action space.

[0028] This embodiment proposes a three-stage submodal Gaussian process framework, SubGP, to fundamentally solve the long-tail preference problem. In the batch Thompson sampling stage, an adaptive covariance scaling factor and a dual-independent sampling mechanism are used to conduct extensive exploration in the early stages and gradually focus on development in the later stages, generating diverse candidate preference pairs. In the submodal marginal gain evaluation stage, the diminishing returns property of the submodal function is utilized to quantify the information value of each preference pair. By maximizing the marginal gain, the most informative preference pair is selected, effectively suppressing the dominant effect of head preferences. In the Gaussian process posterior update stage, the observed preferences are integrated into a Bayesian framework, refining the posterior distribution to guide subsequent sampling. This framework significantly reduces the number of preference queries required by optimizing the preference learning process, improving learning efficiency and accuracy, and providing more reliable preference learning support for human-computer collaborative systems.

[0029] These three phases enable SubGP to effectively mitigate the impact of long-tail preference relationships, while ensuring efficient identification of the global optimum through a principled balance between exploration and development. Furthermore, SubGP significantly reduces the number of user preference queries required, making it particularly advantageous in high-dimensional environments.

[0030] The specific process of each step in the above embodiments will be described below.

[0031] This invention reveals, through theoretical analysis, the impact of long-tail preference relationships on pairwise comparison-based preference learning paradigms. This application finds that this preference bias essentially causes existing algorithms to converge to suboptimal local solutions. To address this issue, this application introduces a new optimization dimension into Gaussian process (GP) preference learning within the Bayesian optimization framework. This application proposes using a submodular set function to capture diminishing returns on preference pairs, forming a new algorithm—the Submodular Gaussian Process (SubGP).

[0032] S1, Initialization, setting up motion space Gaussian process prior covariance Noise parameters Maximum number of iterations and batch size Initialize the preference set Access set Potential function value dictionary and Gaussian process posterior .

[0033] S2 specifically includes the following processes: S21, Batch Thompson Sampling: Calculate the adaptive covariance scaling factor, and generate a set of candidate preference pairs based on the Gaussian process posterior of the previous iteration, the batch size, and the covariance scaling factor using a dual independent sampling mechanism.

[0034] Unlike traditional pairwise GP methods that sample only a single pair of comparisons per iteration, the SubGP in this application employs batch Thompson sampling to generate a diverse set of candidate preference pairs, containing multiple candidate pairs. This is based on the need to accelerate action space exploration and reduce redundancy, which is a limitation observed in standard pairwise GP learning. The principle of batch Thompson sampling has been theoretically proven to significantly improve efficiency while achieving a regret bound comparable to sequential sampling.

[0035] To achieve a dynamic balance between exploration and development, this application introduces a covariance scaling factor. ,in Indicates iterative index, It is the initial scaling factor. It's the attenuation rate. This adaptive scaling allows SubGP to conduct extensive exploration in the early stages (larger attenuation rates). (value), and then gradually focus on development (smaller) as iterations progress. Specifically, the covariance scaling factor controls sampling uncertainty by adjusting the posterior covariance matrix. In the iteration... When the scaled covariance matrix is ,in It is the original posterior covariance matrix. A larger one... The higher the value, the more random the sampling becomes, promoting a broader exploration of the action space; a smaller value... This reduces uncertainty and allows sampling to be more focused on high-value areas, enabling development.

[0036] Furthermore, to ensure diversity within each batch, SubGP employs a double independent sampling scheme instead of simply splitting a single posterior sample. For each candidate preference pair, two independent samples are taken from the posterior distribution, rather than using different parts of the same sample, ensuring independence between candidate pairs. During iterations... At that time, from the posterior model Mid-sampled latent function value ,in Indicates the first Each sampling is independent. Select the action with the largest sample value. Then, the actions selected from the two independent samples and Composition of candidate preference pairs .

[0037] This method reduces the correlation between samples, enabling more efficient exploration of different regions in the preference space. In iterations... At that time, the process generates a candidate set. Each iteration includes diverse preference actions, where the index... and Used as a reference for tracking actions in the original space. The generated batches provide a rich and diverse set of candidate comparisons, laying the foundation for a balanced explore-explore approach. By leveraging this parallel sampling method, SubGP achieves three advantages: (i) improved sampling efficiency and reduced redundancy; (ii) greater robustness to the dominance of head preferences in long-tailed distributions; and (iii) faster and more reliable convergence in high-dimensional preference learning tasks.

[0038] S22, Submodular Marginal Gain Evaluation: Based on the candidate preference pair set, access set, and latent function value dictionary, the information value of each candidate preference pair is quantified through the submodular evaluation function, the marginal gain of the candidate preference pair relative to the selected set is calculated, and the candidate preference pair that maximizes the marginal gain is selected to form the optimal subset.

[0039] The submodular marginal gain evaluation establishes a systematic framework for quantifying the informational value of candidate preference pairs. Specifically, this application first defines the submodular evaluation function. We assess the utility of preference pairs in the selection process and then develop an optimization method through marginal gain analysis to effectively handle non-additive pairwise relationships while reducing variance in the optimization process.

[0040] Given preference pairs Access Collection and the dictionary of potential function values Define the submodulus evaluation function :

[0041] This function assigns higher values ​​to preference pairs that represent unexplored regions in the action space or contribute more to the preference learning process. Specifically, if neither action in a preference pair has been evaluated, the pair receives the highest score of 10 because it can provide new information about unknown regions. If one action has been evaluated and the other has not, and the latent function value of the unevaluated action is higher than that of the evaluated action, the preference pair also receives a score of 10 because it may reveal a better choice. In other cases, a score of 0 indicates that the preference pair has low informational value. Represents the set of evaluated points, from the preference set. It is formed by the aggregation of all observation points in the middle, and Store the latent function value for each evaluated point. Candidate preference set Generated through a batch Thompson sampling process.

[0042] Given a submodular evaluation function Preference pair relative to the selected set The marginal gain is:

[0043] Marginal gain was precisely measured. Add to The incremental value. Marginal gain reflects the diminishing returns property of the submodular function: as more preferences are selected... The marginal contribution of newly added preference pairs gradually decreases. This characteristic ensures that the algorithm prioritizes preference pairs with the most information while avoiding redundancy. Following the submodular function property, preference pairs are selected through iterative updates:

[0044] in, Indicates all candidate preference pairs, Indicates the current selection. It is the optimal preference pair that maximizes marginal gain.

[0045] The submode marginal gain evaluation process is divided into three stages: basic evaluation, continuous pair detection, and marginal gain maximization selection.

[0046] In the basic evaluation phase, the algorithm evaluates each candidate point pair. latent function value and Then, based on the function values, the strengths and weaknesses are identified, and the submodular function is used. Calculate the initial score.

[0047] In the successive pair detection phase, the algorithm identifies preference pair chains with transitive relational structures (e.g., and They are assigned a medium priority to facilitate efficient exploration. Specifically, the algorithm iterates through all preference pairs, checking for the existence of pairs of the form... and consecutive pairs, of which Preferences appear simultaneously in two preference pairs. For identified chains of consecutive pairs, the score of the first preference pair in the chain is adjusted to 5 (medium priority). This is done because consecutive pairs provide transitive information, which helps in constructing a complete preference ranking, but their information gain is relatively small compared to completely unexplored preference pairs. Medium priority ensures that these preference pairs are selected at the appropriate time, promoting efficient exploration. By prioritizing consecutive pairs, the algorithm can leverage the transitivity of preference relationships, reducing the number of direct comparisons required.

[0048] In the marginal gain maximization selection phase, the algorithm sorts candidate pairs by score and then iteratively selects the maximum gain. If no positive marginal gain is found during the iteration process, the preference pair with the highest score is selected. This ensures that the selected set collectively provides the maximum information gain.

[0049] This evaluation mechanism effectively addresses the long-tail preference problem by simultaneously prioritizing unexplored areas and potential superior alternatives, while minimizing variance through marginal gain decomposition.

[0050] S23, Preference Query and Data Update: Perform a human preference query on each preference pair in the optimal subset to obtain preference relationships and update the preference set, access set, and latent function value dictionary.

[0051] S24, Gaussian process posterior update: Based on the updated preference set, a regularized Bayesian update method is used to integrate observed preferences to refine the Gaussian process posterior, resulting in a posterior model containing the posterior mean, covariance eigenvector, covariance eigenvalue, and optimal action index.

[0052] Specifically, in order to integrate The observed preferences are updated using a regularized Bayesian approach in SubGP. The adaptive covariance structure reflects the increasing certainty of the latent preference function, a refined posterior. This guides subsequent sampling iterations.

[0053] Represent the latent function as a vector. The prior distribution is assumed to be a multivariate Gaussian distribution:

[0054] in, It is the covariance matrix specified as the kernel function. The likelihood function is defined as:

[0055] This likelihood function guarantees consistency between the latent function value and the preference relationship. A scaling parameter is introduced. To allow for tolerance of noise. The sigmoid function is defined as follows: The posterior function is ultimately obtained as:

[0056] The posterior mean is obtained by minimizing the following objective function:

[0057] in, The loss function for each comparison is a submodulus function, used here to characterize the preference relationship. Unlike the traditional pairwise GP formula, the formula in this application explicitly mitigates the influence of long-tail preference relationships, thereby enhancing the robustness of preference learning.

[0058] The posterior mean is the solution to the above optimization problem. If the covariance scaling parameter is greater than zero, the eigenvectors and eigenvalues ​​of the covariance matrix are computed. The inverse of the posterior covariance approximation is obtained by evaluating the Hessian matrix of the objective function at the posterior mean estimate:

[0059] Then, the eigenvalues ​​and eigenvectors of the inverse posterior covariance matrix are calculated. To ensure numerical stability, a small regularization term is added to the covariance inverse matrix. The negative eigenvalues ​​are then set as the minimum positive values. Finally, the eigenvalues ​​corresponding to the covariance matrix are obtained by inverting the eigenvalues. The returned model posterior contains the following information: posterior mean. Eigenvectors of the covariance matrix, eigenvalues ​​of the covariance matrix, and the index of the best action (the index corresponding to the maximum posterior mean).

[0060] The SubGP process begins with batch Thompson sampling, generating a strategically diverse set of candidate preference pairs. These candidates are then rigorously evaluated using submodular marginal gain assessment, selecting only the most informative subset for preference queries. The loop is completed via Bayesian updates, where the observed preferences are refined into a posterior distribution to produce latent preference function values. Increasingly accurate estimates.

[0061] The real difference with SubGP lies in its ability to handle preference bias. The algorithm innovatively decomposes diminishing returns into measurable marginal gains in preferences, allowing for effective management of non-additive pairwise relationships while maintaining strong generalization performance.

[0062] Please refer to Figure 4 , Figure 4 This presents a performance comparison between SubGP and traditional Pairwise GP in a constrained linear bandit environment. Figure 4 (a) shows that as the number of arms increases from 10 to 300, the number of convergence iterations required by SubGP is consistently significantly lower than that of PairwiseGP and closer to the Oracle benchmark. Figure 4(b) demonstrates the convergence process of the preference values: the true optimum is 0.32114. PairwiseGP stalls at a suboptimal gap of approximately 0.002 during iterations, ultimately converging only to 0.31896, while SubGP consistently and stably approaches the global optimum and eventually reaches the true optimum of 0.32114. This verifies that SubGP effectively overcomes the weakness of PairwiseGP, which is prone to getting trapped in local optima under biased preference distributions. Experimental verification confirms the superior efficiency of SubGP, consistently reducing the required preference queries compared to baseline methods—especially significant in high-dimensional environments filled with irrelevant features. This efficiency advantage becomes particularly important in practical applications where human preferences represent scarce or expensive resources.

[0063] The complete process of the SubGP algorithm is as follows: (1) Initialization: Set up the action space GP prior covariance Noise parameters Maximum number of iterations Batch size (Number of candidate preference pairs generated in each iteration).

[0064] (2) Initialize the preference set Access set Potential value dictionary .

[0065] (3) Set GP posterior .

[0066] (4) : Step 1: Calculate covariance scaling ; Step 2: Perform batch Thompson sampling ; Step 3: Perform sub-mode marginal gain evaluation ; Step 4: For Each preference pair in : Query preferences ,get (in Indicates the index of the winning side. (Index indicating the losing side); renew ; renew ; renew ; Step 5: Use Post-update verification .

[0067] (5) Return to the optimal action .

[0068] In summary, this invention discloses a human-computer collaboration method, SubGP, to address human bias, aiming to effectively extend preference learning to high-dimensional action space scenarios. In the first stage of SubGP, this application proposes a batch Thompson sampling algorithm, which achieves a dynamic balance between exploration and exploitation through adaptive covariance scaling and a dual independent sampling mechanism, significantly improving sampling efficiency and reducing redundancy. In the second stage, this application introduces submodulus marginal gain evaluation to quantify the information value of preference pairs, effectively addressing the long-tail preference relationship problem. In the third stage, this application employs regularized Bayesian updates to integrate observed preferences. This invention verifies the potential of SubGP in the field of preference learning, significantly reducing the number of preference queries required, improving learning efficiency and accuracy, and providing more reliable preference learning support for human-computer collaboration systems.

[0069] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A human-computer collaboration method for addressing human biases, characterized in that, include: S1, Initialization: Set the action space, Gaussian process prior covariance, noise parameters, maximum number of iterations and batch size; initialize the preference set, access set, latent function value dictionary and Gaussian process posterior. S2, iteratively execute batch Thompson sampling, batch Thompson sampling, preference query and data update, and Gaussian process posterior update until the maximum number of iterations is reached: S21, Batch Thompson Sampling: Calculate the adaptive covariance scaling factor, and generate a set of candidate preference pairs based on the Gaussian process posterior of the previous iteration, the batch size, and the covariance scaling factor using a dual independent sampling mechanism. S22, Batch Thompson Sampling: Based on the candidate preference pair set, access set and latent function value dictionary, the information value of each candidate preference pair is quantified by the submodulus evaluation function, the marginal gain of the candidate preference pair relative to the selected set is calculated, and the candidate preference pairs that maximize the marginal gain are selected to form the optimal subset; S23, Preference Query and Data Update: Perform human preference query on each preference pair in the optimal subset to obtain preference relationships and update the preference set, access set and latent function value dictionary; S24, Gaussian process posterior update: Based on the updated preference set, a regularized Bayesian update method is used to integrate observed preferences to refine the Gaussian process posterior, resulting in a posterior model containing the posterior mean, covariance eigenvector, covariance eigenvalue, and optimal action index. S3, after the iteration is complete, returns the optimal action corresponding to the maximum potential function value in the action space.

2. The human-machine collaboration method for addressing human bias according to claim 1, characterized in that, In S21, the dual independent sampling mechanism of batch Thompson sampling is as follows: two independent samplings are performed from the posterior model to obtain the latent function value. The action with the largest latent function value in the two samplings is selected to form a candidate preference pair. This process is repeated until K candidate preference pairs are generated.

3. The human-machine collaboration method for addressing human bias according to claim 1, characterized in that, Covariance scaling factor in S21 The expression is as follows: in, It is the initial scaling factor. It is the attenuation rate. Indicates the iterative index.

4. The human-machine collaboration method for addressing human bias according to claim 1, characterized in that, Modular evaluation function in S22 G The expression is as follows: in, For two actions in a candidate preference pair For action The corresponding latent function value, For access set.

5. The human-machine collaboration method for addressing human bias according to claim 4, characterized in that, Preference pair relative to the selected set The marginal gain is: in, For the currently selected preference pair set, For candidate preference pairs, For set The sum of the submodular evaluation function values ​​of all preference pairs.

6. The human-machine collaboration method for addressing human bias according to claim 1, characterized in that, The submode marginal gain evaluation also includes a successive pair detection step: Identify preference pair chains with transitive relational structures, and adjust the submodular evaluation score of the first preference pair in the chain to medium priority to balance exploration efficiency and information gain.

7. The human-machine collaboration method for addressing human bias according to claim 1, characterized in that, In S24, during the Gaussian process posterior update, the posterior mean is obtained by minimizing the objective function. Defined as: Among them, fractional difference , and Representing candidate solutions respectively and The potential function value, The covariance matrix of the Gaussian process is... For the log-likelihood term based on the submodular function, is the vector of potential functions.

Images