A combined enhanced ternary delay human-machine collaboration method

Abstract

The application relates to the technical field of artificial intelligence, in particular to a combined enhanced ternary delay man-machine cooperation method, which comprises the following steps: receiving an input sample to be processed; determining at least three candidate decision modes for the input sample; determining an optimal decision mode from the three candidate decision modes based on the characteristics of the input sample through an adaptive selection module; wherein the adaptive selection module is used for modeling the mode selection as a ternary decision problem to adaptively match the optimal decision mode most suitable for the current sample characteristics based on the ternary decision problem; and executing the determined optimal decision mode to output a final decision result for the input sample. The method can adaptively select the optimal path among the three decision modes, and eliminate noise through human-guided conformal prediction of model prediction, so that the overall classification accuracy of the man-machine cooperation system is significantly improved.

Description

Technical Field

[0001] The embodiments of this application relate to the field of artificial intelligence technology, and in particular to a combined enhanced ternary delay human-machine collaboration method. Background Technology

[0002] In recent years, artificial intelligence has made significant progress in many fields, but its reliability still faces challenges in complex real-world tasks. Human-machine collaboration, by integrating the complementary strengths of humans and AI, has become an effective solution. Among these approaches, the Learning to Defer framework is one of the mainstream methods, allowing the system to delegate decision-making tasks to humans under specific circumstances.

[0003] However, existing human-machine collaboration methods generally suffer from the following shortcomings: First, most methods are limited to rigid binary choices, such as choosing only between "AI or human" or only between "AI or human-machine combination". This limitation cannot adapt to complex scenarios in real-world applications where AI-only, human-only, and human-machine combination modes are all indispensable, making it difficult to fully unleash the system's potential. Second, when the model provides a low-confidence prediction for a sample, and the human judgment is correct, handing the sample over to the human-machine combination mode can lead to the uncertainty noise introduced by the model, which can actually damage the combination performance, resulting in combination degradation, meaning the combination result is worse than relying solely on humans. Furthermore, ternary mode selection and fusion methods have limitations. The few existing works exploring ternary paradigms either fail to capture fine-grained distribution features by simply splicing human labels and model probabilities, or rely on expensive concept annotations, limiting their practicality; and face significant challenges in distinguishing between "human-only" and "combination" modes, because human confidence is usually an implicit variable that is difficult to obtain. Summary of the Invention

[0004] In view of this, embodiments of this application propose a combined enhanced ternary delayed human-machine collaboration method, which aims to construct an adaptive human-machine collaboration decision-making mechanism, break through the binary limitation, realize accurate dynamic selection of three modes: AI independent prediction, human independent prediction, and human-machine combination, and effectively suppress the performance degradation of the combination mode caused by unreliable model prediction. In this way, it can adaptively select the optimal path among the three decision modes, and eliminate noise by performing human-guided conformal prediction on the model prediction, thereby significantly improving the overall classification accuracy of the human-machine collaboration system.

[0005] To achieve the above objectives, embodiments of this application propose a combined enhanced ternary delay human-machine collaboration method, the method comprising the following steps: Receive the input sample to be processed; Identify at least three candidate decision-making modes for the input sample; wherein the three candidate decision-making modes include an AI independent decision-making mode, a human independent decision-making mode, and a human-machine combined decision-making mode after enhanced fusion processing; The adaptive selection module determines the optimal decision mode from three candidate decision modes based on the features of the input sample. The adaptive selection module is used to model the mode selection as a ternary decision problem, so as to adaptively match the optimal decision mode that best suits the features of the current sample based on the ternary decision problem. Execute the determined optimal decision-making model to output the final decision result for the input sample.

[0006] This application proposes a combined enhanced ternary delayed human-machine collaboration method. First, it receives an input sample to be processed. Then, it determines at least three candidate decision modes for the input sample, including an AI independent decision mode, a human independent decision mode, and a human-machine combined decision mode after enhanced fusion processing. Next, an adaptive selection module determines the optimal decision mode from the three candidate decision modes based on the features of the input sample. The adaptive selection module models the mode selection as a ternary decision problem, adaptively matching the optimal decision mode best suited to the current sample features based on the ternary decision problem. Finally, it executes the determined optimal decision mode to output the final decision result for the input sample. This solution constructs an adaptive human-machine collaborative decision-making mechanism that overcomes the binary constraint, achieving accurate dynamic selection of three modes: AI independent prediction, human independent prediction, and human-machine combined prediction. It effectively suppresses the performance degradation of combined modes caused by unreliable model predictions, thus adaptively selecting the optimal path among the three decision modes and eliminating noise through human-guided conformal prediction of model predictions, thereby significantly improving the overall classification accuracy of the human-machine collaborative system.

[0007] Optionally, an adaptive selection module determines the optimal decision mode from three candidate decision modes based on the features of the input sample, including: calculating the first score of the input sample corresponding to the AI ​​independent decision mode; wherein the first score is a non-parametric score of the model's intrinsic confidence; collaboratively generating a second score of the input sample corresponding to the human independent decision mode and a third score corresponding to the human-machine combined decision mode through a discriminative gating network; wherein the second score and the third score are parametric scores; and determining the decision mode with the highest score as the optimal decision mode by comparing the first score, the second score, and the third score.

[0008] Optionally, the intrinsic confidence of the model is the log difference between the highest probability and the second highest probability output by the model, and its expression is as follows: ; in, Indicates the first score; Represents the logarithm of the highest probability; This represents the logarithm of the second highest probability.

[0009] Optionally, the discriminative gating network is trained by jointly optimizing the routing loss and the ranking loss; the routing loss is used to maximize the probability of the optimal decision pattern being selected; the ranking loss is used to forcibly correct the relative ranking relationship between the second and third scores, so as to give higher scores to the human independent decision pattern in the event of combinatorial degradation.

[0010] Optionally, the routing loss is calculated by dynamically assigning weights to each decision mode; wherein, the weight of the AI ​​independent decision mode is a binary indicator of its prediction correctness, and the weights of the human independent decision mode and the human-machine combined decision mode are jointly determined by a differentiable function guided by model uncertainty, and an evaluation term based on the human interaction cost and the penalty term for prediction error.

[0011] Optionally, the ranking loss uses the Softplus function to smooth out score inversions where the second score is lower than the third score.

[0012] Optionally, the human-machine combined decision-making model further includes: obtaining hard labels given by human experts for the input samples, and the initial probability vector output by the pre-trained model for the input samples; mapping the hard labels to human soft label vectors; refining the initial probability vectors through human-guided conformal prediction to filter out low-confidence categories, obtaining the refined probability vectors; calculating Bayesian combination probabilities for all categories based on the human soft label vectors and the refined probability vectors, calculating the combination probabilities through Bayesian fusion, and determining the category with the highest probability as the combined decision result.

[0013] Optionally, the initial probability vector is refined through human-guided conformal prediction to filter out low-confidence categories, resulting in a refined probability vector, including: Calculate the inconsistency score incorporating human priors; the formula includes: ; in, Indicates non-consistent scores; The model represents the categories The predicted probability; Indicates human perception of categories The probability of soft labels; This represents the parameters used to balance the model's weights with human input. Based on the calibration set, the target threshold is determined according to the preset marginal coverage rate, and a prediction set is constructed based on the target threshold; Set the probabilities of all categories other than those in the prediction set to zero in the initial probability vector and normalize them to obtain the refined probability vector.

[0014] Optionally, parameters used to balance model and human weights The value of is dynamically adjusted based on the difficulty level of the samples; where the difficulty level of the samples is determined by comparing the classification abilities of humans and models on each category.

[0015] Optionally, the method provided in this embodiment further includes: dividing the categories into human-easy-to-handle and human-difficult-to-handle groups by comparing the size of the diagonal elements of the human confusion matrix and the model confusion matrix, and configuring optimal parameters for each group. Among them, the human-easy-to-process group and the human-difficult-to-process group do not overlap and together cover the entire category space. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies of this application will be briefly introduced below. Obviously, the following drawings are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. The drawings described herein are only used to explain this application and are not intended to limit this application.

[0017] Figure 1 This is a flowchart of a combined enhanced ternary delay human-computer collaboration method provided in one embodiment of this application; Figure 2 This is a flowchart of the human-machine combination mode execution provided in one embodiment of this application; Figure 3 This is a schematic diagram of a combined enhanced ternary delay human-machine collaboration method provided in one embodiment of this application; Figure 4 This is a schematic diagram illustrating the hardware and software experimental environment provided in one embodiment of this application; Figure 5 This is a schematic diagram illustrating the performance of the experimental evaluation comparison accuracy provided in one embodiment of this application; Figure 6 This is a schematic diagram illustrating the performance of the experimental evaluation comparison accuracy provided in one embodiment of this application; Figure 7 This is a schematic diagram illustrating the experimental evaluation and comparison of performance at different coverage rates provided in one embodiment of this application; Figure 8 This is a schematic diagram illustrating the experimental evaluation of the performance of combination modes under different parameters β in one embodiment of this application; Figure 9 This is a schematic diagram illustrating the experimental assessment of the impact of changes in human capabilities on pattern selection provided in one embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the various embodiments of this application will be described in detail below with reference to the accompanying drawings. Those skilled in the art will understand that many technical details have been presented in the embodiments of this application to facilitate better understanding. However, the technical solutions claimed in this application can be implemented even without these technical details and various variations and modifications based on the following embodiments. The division of the following embodiments is for ease of description and should not constitute any limitation on the specific implementation of this application. The following embodiments can be combined with and referenced by each other without contradiction.

[0019] In recent years, artificial intelligence has made significant progress in many fields, but its reliability still faces challenges in complex real-world tasks. Human-machine collaboration, by integrating the complementary strengths of humans and AI, has become an effective solution. Among these approaches, the Learning to Defer framework is one of the mainstream methods, allowing the system to delegate decision-making tasks to humans under specific circumstances.

[0020] However, existing human-machine collaboration methods generally suffer from the following shortcomings: First, most methods are limited to rigid binary choices, such as choosing only between "AI or human" or only between "AI or human-machine combination". This limitation cannot adapt to complex scenarios in real-world applications where AI-only, human-only, and human-machine combination modes are all indispensable, making it difficult to fully unleash the system's potential. Second, when the model provides a low-confidence prediction for a sample, and the human judgment is correct, handing the sample over to the human-machine combination mode can lead to the uncertainty noise introduced by the model, which can actually damage the combination performance, resulting in combination degradation, meaning the combination result is worse than relying solely on humans. Furthermore, ternary mode selection and fusion methods have limitations. The few existing works exploring ternary paradigms either fail to capture fine-grained distribution features by simply splicing human labels and model probabilities, or rely on expensive concept annotations, limiting their practicality; and face significant challenges in distinguishing between "human-only" and "combination" modes, because human confidence is usually an implicit variable that is difficult to obtain.

[0021] In view of this, embodiments of this application propose a combined enhanced ternary delayed human-machine collaboration method, which aims to construct an adaptive human-machine collaboration decision-making mechanism, break through the binary limitation, realize accurate dynamic selection of three modes: AI independent prediction, human independent prediction, and human-machine combination, and effectively suppress the performance degradation of the combination mode caused by unreliable model prediction. In this way, it can adaptively select the optimal path among the three decision modes, and eliminate noise by performing human-guided conformal prediction on the model prediction, thereby significantly improving the overall classification accuracy of the human-machine collaboration system.

[0022] One embodiment of this application proposes a combined enhanced ternary delay human-computer collaboration method applied to an electronic device, wherein the electronic device can be a terminal or a server. This embodiment and the following embodiments will use a server as an example for description. The implementation details of the combined enhanced ternary delay human-computer collaboration method proposed in this embodiment will be described in detail below. The following implementation details are provided for ease of understanding and are not necessary for implementing this solution.

[0023] Understandably, this embodiment proposes a combined enhanced ternary delayed human-machine collaboration method, which aims to overcome the limitations of the traditional binary rejection mechanism. By introducing three independent decision-making paths (AI independent decision-making, human independent decision-making, and human-machine combined decision-making) and constructing an adaptive routing mechanism, it achieves more accurate and robust human-machine collaboration.

[0024] The specific process of the combined enhanced ternary delay human-computer collaboration method proposed in this embodiment can be described as follows: Figure 1 As shown, it includes: Step 101: Receive the input sample to be processed.

[0025] For example, the system receives an input sample from the external environment that needs to be classified or decided. .

[0026] The sample can be any form of data, such as: a medical image (e.g., an X-ray, CT scan), a natural image (e.g., an image from the ImageNet dataset), a piece of text, a sensor reading, etc.

[0027] In this embodiment, for ease of explanation, ... This will be illustrated using an image classification task as an example. The input space is denoted as... ,sample The tag space is , that is, shared There are several possible categories.

[0028] Step 102: Determine at least three candidate decision modes for the input sample.

[0029] Among them, the three candidate decision-making modes include AI independent decision-making mode, human independent decision-making mode, and human-machine combined decision-making mode with enhanced fusion processing.

[0030] Specifically, the system provides the input samples Three parallel and independent decision paths are pre-defined, i.e., three candidate decision modes. The specific implementations of these three modes are as follows: Mode 1: AI-only Mode (AI Independent Decision-Making Mode) In this mode, the system will rely entirely on pre-trained artificial intelligence models. Make a decision. The specific process is as follows: S11: Input sample Feed into the pre-trained model ,get Predictions for each category ; S12: Convert to a probability vector using Softmax : ; in, ,and . The representative model considers the sample ; Belongs to the The probability of a class.

[0031] S13: Select the category with the highest probability as the AI ​​prediction result: .

[0032] Mode Two: Human-only Mode In this mode, the system will sample The matter is to be independently judged by one or more selected human experts. These experts will base their judgment on their own knowledge and experience. A specific hard label is given from each candidate category. The system directly uses this label as the final human prediction result, and its expression is as follows: ; in, This represents the result of human prediction.

[0033] Mode 3: Human-AI Combination Mode It is worth mentioning that the human-machine combined decision-making model is one of the core innovations of the embodiments of this application, aiming to deeply and robustly integrate the judgment of AI models and human experts. It is achieved through a two-stage enhancement mechanism that combines human-guided conformal prediction with Bayesian fusion to overcome the performance degradation problem caused by unreliable model predictions in traditional combined models.

[0034] In one possible embodiment, the human-machine combined decision-making mode further includes: obtaining hard labels given by human experts for the input samples, and an initial probability vector output by a pre-trained model for the input samples; mapping the hard labels to human soft label vectors; refining the initial probability vectors through human-guided conformal prediction to filter out low-confidence categories, obtaining a refined probability vector; calculating Bayesian combination probabilities for all categories based on the human soft label vectors and the refined probability vectors, calculating the combination probabilities through Bayesian fusion, and determining the category with the highest probability as the combined decision result.

[0035] For example: Figure 2 As shown, the detailed implementation steps of the human-machine collaborative decision-making model are as follows: S31: Human soft tag mapping: First, the system uses a pre-obtained confusion matrix. The hard labels given by human experts Convert to a soft tag vector .

[0036] Wherein, confusion matrix The dimension is ;element Indicates the true category is The samples were classified by human experts. The frequency of soft tag vectors. That is, the hard label in the confusion matrix. The corresponding column, namely: .

[0037] For example, in order to reduce the variance caused by maximum likelihood estimation, this embodiment applies a Dirichlet prior to each column of the confusion matrix for smoothing, thereby improving the stability and reliability of soft tags.

[0038] S32: Human-guided Conformal Prediction Understandably, in order to filter out low-confidence categories in the AI ​​model output that may negatively affect the final fusion result, this embodiment designs a nonconformity score that incorporates human prior knowledge.

[0039] For the sample and a certain hypothesis category Its non-consistency score is defined as: Calculate the inconsistency score incorporating human priors; the formula includes: ; in, Indicates non-consistent scores; The model represents the categories The predicted probability; Indicates human perception of categories The probability of soft labels; This indicates that the model predicts the category to which the sample belongs. The probability of; This represents the parameters used to balance the model and human weights, that is, to balance the relative importance of model probabilities and human confidence scores. . The smaller the value, the more it relies on human judgment.

[0040] For example, parameters used to balance model and human weights The value of is dynamically adjusted based on the difficulty level of the samples; where the difficulty level of the samples is determined by comparing the classification abilities of humans and models on each category.

[0041] Then, based on the calibration set, the target threshold is determined according to the preset marginal coverage rate, and a prediction set is constructed based on the target threshold. The probabilities corresponding to categories other than those in the prediction set in the initial probability vector are set to zero and normalized to obtain the refined probability vector.

[0042] For example: The system utilizes a separate calibration set. To calculate the non-consistency score for all samples, based on a preset significance level. (For example, (Corresponding to a 90% confidence level) Determine a target threshold. This ensures that the calibration set has at least The proportion of samples whose true class corresponds to a non-consistency score is less than or equal to For new samples The system constructs its prediction set: Prediction set It includes all categories with inconsistency scores below a threshold, and its expression is as follows: ; Subsequently, the system excluded the prediction set. In addition to the low-confidence categories, a refined model probability vector is obtained. And use it to update the original model output: This process effectively filters out unreliable parts of the model's predictions.

[0043] S33: Sample Grouping For example, by comparing the diagonal elements of the human confusion matrix with those of the model confusion matrix, categories are divided into human-easy-to-handle and human-difficult-to-handle groups, and optimal parameters are configured for each group. Among them, the human-easy-to-process group and the human-difficult-to-process group do not overlap and together cover the entire category space.

[0044] Specifically, considering the varying degrees of human guidance required for different categories of samples, this embodiment proposes a dynamic grouping strategy based on both human and model capabilities. The system compares the diagonal elements of the human confusion matrix. and the diagonal elements of the model confusion matrix To measure the performance of humans and models in the first quarter Classification ability. If the accuracy of humans in classifying a certain class is not lower than that of the model, then... If so, this category will be classified into the human handling group. Conversely, they are categorized into the group of things that are difficult for humans to handle. : ; ; It should be noted that mutual exclusion partitioning means that each category can only belong to one and must belong to one of the groups, that is, the two groups do not overlap, and their union constitutes the complete label space: ; ; After grouping is completed, the system is in a separate validation set. Above, using a grid search method, respectively and Two sets of methods for finding the optimal weight parameters. and This maximizes the combined accuracy within each group, enabling the system to adaptively adjust the intensity of human guidance based on the characteristics of different categories.

[0045] S34: Bayesian Fusion After the above conformal prediction and grouping optimization, the system uses Bayes' theorem to refine the model probability vector. Compared with human soft tag vectors Perform deep fusion and calculate the combined posterior probability for each category: ; Ultimately, the decision outcome of the human-machine combination model is the category with the highest posterior probability. : .

[0046] Step 103: The optimal decision mode is determined from three candidate decision modes based on the features of the input sample through the adaptive selection module.

[0047] The adaptive selection module is used to model the pattern selection as a ternary decision problem, and adaptively match the optimal decision pattern that best suits the current sample features based on the ternary decision problem.

[0048] Understandably, this step is the core of achieving ternary dynamic decision-making. The system constructs an adaptive selection module, which models pattern selection as a ternary decision problem and, for each sample... Calculate a score vector, corresponding to the applicability scores of the AI-independent mode, the human-independent mode, and the human-machine combined mode. The mode with the highest score will be selected as the optimal decision mode. For example... Figure 3 As shown, samples are dynamically routed to the decision mode with the highest score (i.e., one of the three decision modes mentioned above). It is worth noting that if the human-machine combined decision mode is selected, this embodiment can utilize human-guided conformal prediction to optimize the AI's prediction set and integrate it with human predictions through Bayesian fusion.

[0049] Wherein, the score vector ; The applicability score for the AI-independent mode (i.e., the first score). For the human independent model (i.e., the second score). The applicability score of the human-machine combination pattern (i.e., the third score).

[0050] In one possible embodiment, step 103 includes: calculating a first score for the input sample corresponding to an AI independent decision-making mode; collaboratively generating a second score for the input sample corresponding to a human independent decision-making mode and a third score corresponding to a human-machine combined decision-making mode through a discriminative gating network; and determining the decision-making mode with the highest score as the optimal decision-making mode by comparing the first score, the second score, and the third score.

[0051] The first score is the non-parametric score of the model's intrinsic confidence; the second and third scores are parametric scores.

[0052] First, a hybrid scoring mechanism: (1) AI Independent Mode Score That is, the first score: The first score is a non-parametric score, evaluated directly using the model's intrinsic confidence. Specifically, it uses the log difference between the model's highest and second-highest probabilities: ; in, Indicates the first score; Represents the logarithm of the highest probability; This represents the logarithm of the second highest probability.

[0053] Understandably, the larger the difference, the more confident the model is in its predictions.

[0054] (2) Pattern score of human participation and That is, the first score and the second score: The first and second scores are parameterized using a discriminative gating network consisting of a feature extraction network and linear layers. Collaborative generation, i.e. .

[0055] Understandably, discriminative gating networks Its applicability is dynamically assessed by learning the complex relationship between sample features and the performance of two human participation modes.

[0056] Second, routing decision execution: During the inference test phase, the selected module compares the three scores mentioned above and uses the samples... Directly route to the optimal decision pattern with the highest score : ; For example, for an image that the AI ​​is very confident about, The sample size will be much higher than the other two. It will be routed to AI-independent mode; however, for a sample that even experts find challenging and requires the assistance of AI information... At most, the sample will be routed to the combined mode.

[0057] Third, the gating network training mechanism: In one possible embodiment, the discriminative gating network is trained by jointly optimizing the routing loss and the ranking loss; the routing loss is used to maximize the probability that the optimal decision pattern is selected; the ranking loss is used to force a correction of the relative ranking relationship between the second and third scores so that, in the event of combined degradation, the human independent decision pattern is given a higher score.

[0058] Understandably, this is necessary for training gating networks. In this embodiment, a joint loss function is designed by training through joint optimization of routing loss and ranking loss. ; in, For routing loss, For sorting loss, It is the weighting coefficient that balances the two losses.

[0059] For example, the routing loss is calculated by dynamically assigning weights to each decision mode; where the weight of the AI ​​independent decision mode is a binary indicator of its prediction correctness, and the weights of the human independent decision mode and the human-machine combined decision mode are determined by a differentiable function guided by model uncertainty, and an evaluation term based on the human interaction cost and the penalty term for prediction error.

[0060] Specifically, routing loss The aim is to maximize the probability of the optimal mode being selected. Unlike traditional methods that use static labels, this embodiment dynamically assigns target weights to the three modes. For AI-independent decision-making, the weights are binary indicators that depend on the accuracy of the prediction. For human-independent decision-making and human-machine collaborative decision-making, the weights are controlled not only by a differentiable function guided by model uncertainty but also by evaluation terms reflecting human utility (including basic interaction costs and severe penalties for incorrect predictions). This dynamic weight design enables the gating network to learn more refined and rational routing strategies.

[0061] For example, the ranking loss uses the Softplus function to smooth out score inversions where the second score is lower than the third score.

[0062] Specifically, ranking loss This loss is used to correct the relative ranking relationship between the scores of the "Human Independent Mode" and the "Combined Mode". Its core purpose is to address the combined mode degradation scenario, where the human judgment is correct, but the combined mode fails due to model noise; in this case, the system should prioritize the Human Independent Mode. This loss uses the Softplus function to smoothly penalize the score inversion phenomenon, ensuring that in this specific scenario, Able to exceed This helps to mitigate the risks associated with integration.

[0063] Step 104: Execute the determined optimal decision-making model to output the final decision result for the input sample.

[0064] The optimal decision-making model determined based on the above embodiments The system calls the corresponding mode execution logic and outputs the final decision result. Specifically: like Then the AI ​​will output its independent decision-making result. ; like Then the output will be the result of independent human decision-making. ; like Then the human-machine combination decision-making result will be output. .

[0065] This completes the entire process of processing an input sample, from reception and pattern evaluation to adaptive selection and final decision-making.

[0066] This application proposes a combined enhanced ternary delayed human-machine collaboration method. First, it receives an input sample to be processed. Then, it determines at least three candidate decision modes for the input sample, including an AI independent decision mode, a human independent decision mode, and a human-machine combined decision mode after enhanced fusion processing. Next, an adaptive selection module determines the optimal decision mode from the three candidate decision modes based on the features of the input sample. The adaptive selection module models the mode selection as a ternary decision problem, adaptively matching the optimal decision mode best suited to the current sample features based on the ternary decision problem. Finally, it executes the determined optimal decision mode to output the final decision result for the input sample. This solution constructs an adaptive human-machine collaborative decision-making mechanism that overcomes the binary constraint, achieving accurate dynamic selection of three modes: AI independent prediction, human independent prediction, and human-machine combined prediction. It effectively suppresses the performance degradation of combined modes caused by unreliable model predictions, thus adaptively selecting the optimal path among the three decision modes and purifying noise through human-guided conformal prediction of model predictions, thereby significantly improving the overall classification accuracy of the human-machine collaborative system.

[0067] The specific experimental environment of this embodiment is as follows: Figure 4 As shown. The hardware environment is based on the Linux operating system Ubuntu 18.04.6, utilizing an NVIDIA GeForce RTX 3090 graphics card with 24.0 GiB of video memory for accelerated computing. The software is based on the GPU-based deep learning framework PyTorch, developed using Python 3.8.

[0068] Figure 5The performance of the invention is demonstrated across three datasets. Overall, this embodiment performs excellently in all three scenarios. Specifically, in the general setting, the method provided in this embodiment achieves an improvement of at least 7.17% compared to relying solely on human-independent decision-making or AI-independent decision-making, highlighting the necessity of human-machine collaboration.

[0069] Next, an ablation experiment was conducted in this embodiment to evaluate the human-guided conformal prediction and adaptive selection modules. First, the human-guided conformal prediction was removed, and instead, the predictions of both humans and AI were directly combined using Bayesian rules. Experimental results show that this ablation operation leads to a significant decrease in accuracy, especially on the CIFAR-100 dataset, where the decrease reaches 2.33%.

[0070] Furthermore, the method provided in this embodiment is superior to CP-CM. Since AI-only, human-only, and CP-CM constitute the three basic decision-making modes in the method provided in this embodiment, it proves that using any single mode is not optimal; therefore, this embodiment should adaptively select among these three modes. Secondly, for the adaptive selection module, this embodiment employs a threshold-based strategy: if the model confidence exceeds... The sample will be routed to "AI only" mode; if the confidence level is lower than If the target value is not found, the route is sent to the "human-only" mode; otherwise, it is sent to the combined mode. This ablation operation leads to a significant drop in accuracy, highlighting the importance of adaptive selection. Intuitively, processing samples by the combined mode seems to outperform processing by humans only, as it integrates human knowledge with model predictions. However, this conclusion does not always hold true. Specifically, in the "human-superior" setting, results show that C2D (choosing between AI or the combined mode) typically performs worse than baseline methods like "Learning to Defer," such as two-stage deferral (choosing between AI or humans). This counterintuitive result suggests that in some scenarios, relying solely on humans may yield better results. Therefore, including all three modes is crucial.

[0071] like Figure 6 As shown, Figure 6This embodiment demonstrates its performance at different coverage levels. It analyzes the relationship between accuracy and coverage (i.e., the proportion of samples predicted by AI) to evaluate system performance under limited human resources. Notably, the need for human resources decreases as coverage increases. An accuracy-coverage curve is generated by ranking test samples using deferral scores and varying the coverage between 0 and 1 using a scanning threshold. In the method provided in this embodiment, the deferral score is defined as the maximum of the "purely human score" and the "combined score," and each sample is assigned to the pattern that reaches this maximum. As shown in the figure, the method provided in this embodiment outperforms the baseline method in accuracy at most coverage levels. Furthermore, when the coverage is 0 (i.e., the model is not involved in prediction at all), the method provided in this embodiment (human or combined) outperforms the C2D method (combined). This indicates that assigning all samples to combined patterns is not always the optimal strategy.

[0072] like Figure 7 As shown, Figure 7 This application demonstrates different parameters The performance of the combined mode. This embodiment studies the parameters. The impact of combined patterns on accuracy across sample groups of different difficulty levels (i.e., easy and hard). It is worth noting that when... At that time, the method provided in this embodiment degenerates into standard conformal prediction. As shown in the figure, the "human-guided conformal prediction" proposed in this application (marked with an asterisk, representing the optimal method when the combined accuracy reaches its peak) The value consistently outperformed the standard CP in all experimental settings. This demonstrates the effectiveness of integrating human guidance. The Easy Group outperformed the Hard Group. The value is low. Because... The human weighting supports the assumption in this embodiment that simple samples require more reliance on human input. Notably, a significant accuracy gap is observed between the easy and difficult groups on the Chaoyang dataset. This is because humans exhibit near-perfect (over 98%) predictive ability across both categories, significantly boosting the accuracy of the easy group.

[0073] like Figure 8 As shown, Figure 8This paper presents the precision, recall, and F1 score for each mode across three datasets. To evaluate the rationality of the mode selection method provided in this embodiment, the ground truth is first defined: if the "AI-only" mode predicts correctly or all modes fail to predict, it is prioritized as the ground truth; otherwise, it is assigned to the successfully predicted "AI-only" or "Combination" modes. Subsequently, precision, recall, and F1 score are used to evaluate the assignment performance. Figure 8 As shown, the method provided in this embodiment achieved high F1 scores on all three datasets. This result demonstrates that the method provided in this embodiment can accurately assign samples to their most suitable modes. Unlike the human-machine combined mode, the "AI only" and "human only" modes exhibit high precision but low recall. This indicates that the system adopts a cautious strategy, defaulting to processing ambiguous samples by the combined mode.

[0074] like Figure 9 As shown, Figure 9 This application demonstrates the method provided to assess the impact of changes in human capabilities on pattern selection. This is achieved by examining the impact of different numbers of predictable human categories. Under these circumstances, how do the selection ratios of the three modes change? A fixed ResNet-18 model was used on the CIFAR-100 dataset, and nine "humans" with different levels of expertise were synthesized. As shown in the figure, as human abilities gradually improve, the selection ratio of the pure human-only mode increases, while the selection ratio of the pure AI-only mode decreases. The accuracy of the pre-trained ResNet-18 model is approximately 72%. When the accuracy reaches 14 (corresponding to approximately 70%), the abilities of humans and models become comparable, at which point the selection rate of Combination Mode reaches its highest level.

[0075] In summary, compared with the prior art, the method provided by the embodiments of this application has the following technical effects: First, this application proposes a combined enhanced ternary human-machine collaborative decision-making paradigm (TriHAI), which breaks through the limitations of the traditional binary rejection mechanism and formally expands the decision space to include three modes: AI independent decision-making, human independent decision-making, and human-machine combined decision-making. This enables it to adapt more flexibly and comprehensively to complex and diverse real-world application scenarios.

[0076] Second, this application proposes an adaptive mode selection strategy based on a hybrid scoring mechanism. By dynamically comparing the model's intrinsic confidence with the parameterized evaluation score generated by a discriminative gating network, the system can effectively distinguish the applicable boundaries of human independent modes and combined modes, accurately route and match the most suitable decision mode for each input sample, and significantly improve the rationality and accuracy of sample allocation.

[0077] Third, this application proposes a human-guided conformal prediction-enhanced fusion mechanism. This mechanism effectively filters and suppresses probabilistic noise introduced by unreliable model predictions into the combined patterns by combining human soft labels to calculate inconsistency scores and construct a prediction set. This significantly improves the accuracy of human-machine fusion decision-making while providing theoretical guarantees of marginal coverage.

[0078] Fourth, extensive experiments on three datasets (ImageNet-16H, Chaoyang, and CIFAR-100) demonstrate that the proposed TriHAI method comprehensively surpasses the state-of-the-art human-machine collaborative benchmark methods in terms of overall classification accuracy, with the highest improvement reaching 9.57%.

[0079] The steps described above are for clarity only. In implementation, they can be combined into one step, or some steps can be broken down into multiple steps, as long as they involve the same logical relationship, they are all within the scope of protection of this application. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, without changing the core design of the algorithm and process, are also within the scope of protection of this application.

[0080] Those skilled in the art will understand that the above embodiments are specific implementations of this application, and in practical applications, various changes can be made in form and detail without departing from the spirit and scope of this application. For those skilled in the art, several improvements and modifications can be made without departing from the principles of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. A combined enhanced ternary delay human-machine collaboration method, characterized in that, The method includes: Receive the input sample to be processed; Identify at least three candidate decision-making modes for the input sample; wherein the three candidate decision-making modes include an AI independent decision-making mode, a human independent decision-making mode, and a human-machine combined decision-making mode after enhanced fusion processing; The adaptive selection module determines the optimal decision mode from three candidate decision modes based on the features of the input sample. The adaptive selection module is used to model the mode selection as a ternary decision problem, so as to adaptively match the optimal decision mode that best suits the features of the current sample based on the ternary decision problem. Execute the determined optimal decision-making model to output the final decision result for the input sample.

2. The method according to claim 1, characterized in that, The step of determining the optimal decision mode from three candidate decision modes through the adaptive selection module based on the features of the input sample includes: Calculate the first score of the input sample corresponding to the AI ​​independent decision-making pattern; where the first score is the non-parametric score of the model's intrinsic confidence. By using a discriminative gating network, a second score corresponding to the human independent decision-making mode and a third score corresponding to the human-machine combined decision-making mode are collaboratively generated for the input samples; wherein, the second score and the third score are parameterized scores. By comparing the first, second, and third scores, the decision pattern with the highest score is determined as the optimal decision pattern.

3. The method according to claim 2, characterized in that, The intrinsic confidence of the model is the logarithmic difference between the highest probability and the second highest probability output by the model, and its expression is as follows: ； in, Indicates the first score; The logarithm of the highest probability; This represents the logarithm of the second highest probability.

4. The method according to claim 2, characterized in that, Discriminative gating networks are trained by jointly optimizing routing loss and ranking loss; The routing loss is used to maximize the probability that the optimal decision pattern is selected; Ranking loss is used to force a correction of the relative ranking relationship between the second and third scores, so that when combinatorial degradation occurs, higher scores are given to the human independent decision-making pattern.

5. The method according to claim 4, characterized in that, The routing loss is calculated by dynamically assigning weights to each decision mode; the weight of the AI ​​independent decision mode is a binary indicator of its prediction correctness, while the weights of the human independent decision mode and the human-machine combined decision mode are determined by a differentiable function guided by model uncertainty, and an evaluation term based on the human interaction cost and the penalty term for prediction error.

6. The method according to claim 4, characterized in that, The ranking loss uses the Softplus function to smooth out the score inversion phenomenon where the second score is lower than the third score.

7. The method according to claim 1, characterized in that, The human-machine combined decision-making mode further includes: Obtain the hard labels given by human experts for the input samples, as well as the initial probability vector output by the pre-trained model for the input samples; Map hard labels to human soft label vectors; The initial probability vector is refined by human-guided conformal prediction to filter out low-confidence categories, resulting in a refined probability vector. Based on the human soft label vector and the refined probability vector, Bayesian combination probabilities are calculated for all categories, and the combination probabilities are calculated through Bayesian fusion. The category with the highest probability is determined as the combination decision result.

8. The method according to claim 7, characterized in that, The human-guided conformal prediction refines the initial probability vector to filter out low-confidence categories, resulting in a refined probability vector, including: Calculate the inconsistency score incorporating human priors; the formula includes: ； in, Indicates non-consistent scores; The model represents the categories The predicted probability; Indicates human perception of categories The probability of soft labels; This represents the parameters used to balance the model's weights with human input. Based on the calibration set, the target threshold is determined according to the preset marginal coverage rate, and a prediction set is constructed based on the target threshold; Set the probabilities of all categories other than those in the prediction set to zero in the initial probability vector and normalize them to obtain the refined probability vector.

9. The method according to claim 8, characterized in that, The parameters used to balance model and human weights The value of is dynamically adjusted based on the difficulty level of the samples; where the difficulty level of the samples is determined by comparing the classification abilities of humans and models on each category.

10. The method according to claim 9, characterized in that, The method further includes: By comparing the diagonal elements of the human confusion matrix and the model confusion matrix, the categories are divided into a human-easy-to-handle group and a human-difficult-to-handle group, and optimal parameters are configured for each group. Among them, the human-easy-to-process group and the human-difficult-to-process group do not overlap and together cover the entire category space.

Images