Man-machine computing task allocation method based on confidence coefficient and cost
Through the human-machine task allocation method based on confidence and cost, the executor of image classification tasks is judged, and the problems of resource redundancy and cost increase in the existing technology are solved, efficient task allocation and resource utilization are achieved, and human-machine is improved. The overall benefits of collaboration.
Patent Information
- Application Number
- CN202411076939.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-12-06
AI Technical Summary
The existing human-machine task allocation methods have problems of resource redundancy and increased costs, and lack dynamic response capabilities, which fail to fully utilize the advantages of human-machine collaboration.
The human-computer calculation task allocation method based on confidence and cost. By obtaining the confidence of humans and machine performers, setting confidence thresholds, judging the executor of image classification tasks, and determining that the task is completed by humans, machines or humans and machines. Optimize resource utilization efficiency.
It improves the accuracy of task completion and resource utilization efficiency, reduces the investment in human and machine costs, and realizes flexible task allocation and intelligent scheduling in a dynamic environment.
Smart Images

Figure CN119088514A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of human-computer computing task allocation, and in particular to a human-computer computing task allocation method based on confidence and cost. Background Art
[0002] In the field of modern technology, the execution of complex tasks often requires the collaboration of humans and machines. Humans and machines are complementary in completing complex tasks, and can combine their respective strengths to improve the efficiency and accuracy of task completion. Existing research shows that people are willing to work with machines, especially when machines can provide skills that humans lack. This cooperation is particularly effective. For example, when the participants' own task capabilities are low and the machine can complete the task more easily, the willingness to cooperate will be higher. Humans and automated systems can collaborate at different levels to improve performance and reduce human burden. At present, many scholars are committed to optimizing the collaborative work of humans and machines in task execution to improve the accuracy and efficiency of task completion. Task allocation between humans and machines has always been a key step in the design of human-machine systems and a major topic in human-computer interaction research. Task allocation is considered an "evergreen" topic and has been covered by a large amount of literature in different fields, including cognitive engineering, human factors, and human-computer interaction.
[0003] At present, existing studies have shown that the accuracy of completing tasks together and fusing results is higher than that of a single person or a single machine. However, existing methods have the following shortcomings: (1) Resource redundancy and increased costs: For each task, both humans and machines need to complete it, which leads to resource redundancy, increased costs and decreased efficiency; (2) Lack of dynamic response capabilities: Traditional human-machine task allocation usually assigns tasks to a single person or a single machine, without considering the situation of human-machine collaboration, thus failing to fully utilize the advantages of human-machine collaboration. Summary of the invention
[0004] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to propose a human-machine computing task allocation method based on confidence and cost. First, it is determined which party, the human executor or the machine executor, has a higher accuracy rate in completing the image classification task. Then, the task allocation system decides whether the task is completed by humans, machines, or humans and machines together according to the confidence and cost evaluation results. The present invention can not only make full use of the advantages of human-machine collaboration and improve the accuracy of task completion, but also flexibly adjust task allocation in a dynamically changing environment and optimize resource utilization efficiency.
[0005] To achieve the above purpose, the technical solution adopted by the present invention is as follows:
[0006] A method for allocating human-computer computing tasks based on confidence and cost, the method comprising:
[0007] Step 1, obtain the confidence of the image classification task, the human performer and the machine performer;
[0008] Step 2, setting a confidence threshold, and obtaining two comparison conditions of confidence levels according to the confidence threshold, the confidence of the human executor, and the confidence of the machine executor;
[0009] Step 3: Based on the two comparison conditions of confidence level, the accuracy of the image classification task performed by the human executor and the machine executor is judged, and the executor of the image classification task is determined;
[0010] The method for determining the executor of the image classification task is:
[0011] When the confidence of the human executor and the confidence of the machine executor meet one of the comparison conditions, and one of the human executor and the machine executor has a higher accuracy rate in performing the image classification task, the image classification task is assigned to the one with higher confidence, that is, the one with higher accuracy rate in the image classification task;
[0012] When the confidence of the human executor and the confidence of the machine executor both meet the two comparison conditions, and the accuracy of the image classification task performed by the human executor and the machine executor is high, the total expected reward of the human executor and the total expected reward of the machine executor are calculated respectively, and the image classification task is assigned to the party with a higher total expected reward, that is, the party with a higher image classification accuracy;
[0013] When the confidence of the human executor and the confidence of the machine executor do not meet the two comparison conditions, the accuracy of the image classification task performed by the human executor and the machine executor is low. The image classification task is assigned to the human executor and the machine executor to complete together, and the accuracy of the image classification task is calculated.
[0014] The confidence of the human executor and the confidence of the machine executor are both obtained from the crowdsourcing platform.
[0015] The two comparison conditions of the confidence level are expressed as:
[0016] η(γ (m,x) ,τ)=w1{|γ (m,x) |>τ}
[0017] η(γ (h,x) ,τ)=w1{|γ (h,x) |>τ},
[0018] Among them, τ is the confidence threshold, γ h is the confidence of the human performer, γ m is the confidence of the machine executor.
[0019] When the confidence of the human executor and the confidence of the machine executor both meet the two comparison conditions, the accuracy of the image classification task performed by the human executor and the machine executor is high, then the total expected reward of the human executor and the total expected reward of the machine executor are calculated respectively, and the image classification task is assigned to the party with the higher total expected reward, that is, the party with the higher image classification accuracy, including:
[0020] Step 2.11, calculate the discrete entropy of the image, expressed as:
[0021]
[0022] Where DE is the discrete entropy of the image, I and J are the number of two-dimensional discrete units of the probability density function, and p i,j is a two-dimensional probability density function;
[0023] The two-dimensional probability density function is expressed as:
[0024]
[0025] Among them, d x represents the derivative kernel in the x direction, d y represents the derivative kernel in the y direction, δ is the Kronecker delta function that describes the binning process in histogram generation, W is the width of the image, and H is the height of the image;
[0026] Step 2.12, calculate the perimeter complexity of the image, expressed as:
[0027]
[0028] Among them, PC is the perimeter complexity of the image, P is the perimeter of the image, and A is the area of the image;
[0029] Step 2.13, use the discrete entropy of the image and the perimeter complexity of the image to calculate the reward after the task is completed, expressed as:
[0030] R(x t )=f1(DE,PC),
[0031] Among them, x t For the current image classification task;
[0032] Step 2.14, calculate the cost of human executors and the cost of machine executors:
[0033] The cost of the human executor is expressed as:
[0034] C h (x t )=f2(DE,PC),
[0035] The cost of the machine executor is expressed as:
[0036] C m (x t )=f3(DE,PC),
[0037] Step 2.15, calculate the total expected reward of the human executor and the total expected reward of the machine executor based on the reward after the task is completed, the cost of the human executor, and the cost of the machine executor:
[0038] The total expected reward for the human performer is expressed as:
[0039]
[0040] The total expected reward of the machine executor is expressed as:
[0041]
[0042] Step 2.16, compare the total expected reward of the human executor and the total expected reward of the machine executor. If the total expected reward of the human executor is greater than the total expected reward of the machine executor, the accuracy of the human executor in performing the image classification task is high, and the image classification task is assigned to the human executor; if the total expected reward of the human executor is less than the total expected reward of the machine executor, the accuracy of the machine executor in performing the image classification task is high, and the image classification task is assigned to the machine executor.
[0043] When the confidence of the human executor and the confidence of the machine executor do not meet the two comparison conditions, the accuracy of the image classification task performed by the human executor and the machine executor is low, the image classification task is assigned to the human executor and the machine executor to complete together, and the accuracy of the image classification task is calculated, specifically:
[0044] Step 2.21, label different categories in the image classification task;
[0045] Step 2.22, use the Bayesian rule to get the conditional probability of each classification label, expressed as:
[0046] p(y|h(xt), m(xt))∝p(h(xt)|y, m(xt))p(y|m(xt)),
[0047] Among them, h(x t ) is the class-level label predicted by the human performer, m(x t ) is the probability level label predicted by the machine executor, and y is the label of the image classification;
[0048] Step 2.23, when both the human performer and the machine performer complete the image classification task independently, the conditional distribution probability of each label is expressed as:
[0049] p(y|h(x t ), m(x t ))∝p(h(x t )|y)p(y|m(x t )),
[0050] Among them, p(h(x t )|y) is the category-level calibration probability when the human performer performs the image classification task independently, p(y|m(x t )) is the instance-level calibration probability when the machine executor performs the image classification task independently;
[0051] Step 2.24, calibrating the probability level labels predicted by the machine executor to obtain the calibrated probability level labels predicted by the machine executor;
[0052] Step 2.25, using the probability-level labels predicted by the calibrated machine performer and the confidence of the human performer, the calibrated probability of the category level when the human performer performs the image classification task independently and the calibrated probability of the instance level when the machine performer performs the image classification task independently are fused to obtain the accuracy of the image classification task completed by the human performer and the machine performer together, expressed as:
[0053]
[0054] Among them, m θ (x) is the probability-level label predicted by the calibrated machine performer, T is the temperature, T∈(0,1) indicates insufficient confidence, and T∈(1,∞) indicates too high confidence.
[0055] A human-machine computing task allocation system based on confidence and cost, comprising:
[0056] The information acquisition module obtains the confidence of the image classification task, the human performer, and the machine performer;
[0057] A comparison condition acquisition module sets a confidence threshold and acquires two comparison conditions of confidence levels according to the confidence threshold, the confidence of the human executor, and the confidence of the machine executor;
[0058] The task assignment module determines the accuracy of image classification tasks performed by human and machine performers based on two comparison conditions of confidence level, and decides who will perform the image classification tasks.
[0059] The method for determining the executor of the image classification task is:
[0060] When the confidence of the human executor and the confidence of the machine executor meet one of the comparison conditions, and one of the human executor and the machine executor has a higher accuracy rate in performing the image classification task, the image classification task is assigned to the one with higher confidence, that is, the one with higher accuracy rate in the image classification task;
[0061] When the confidence of the human executor and the confidence of the machine executor both meet the two comparison conditions, and the accuracy of the image classification task performed by the human executor and the machine executor is high, the total expected reward of the human executor and the total expected reward of the machine executor are calculated respectively, and the image classification task is assigned to the party with a higher total expected reward, that is, the party with a higher image classification accuracy;
[0062] When the confidence of the human executor and the confidence of the machine executor do not meet the two comparison conditions, the accuracy of the image classification task performed by the human executor and the machine executor is low. The image classification task is assigned to the human executor and the machine executor to complete together, and the accuracy of the image classification task is calculated.
[0063] An electronic device, comprising:
[0064] at least one processor;
[0065] and, a memory communicatively coupled to the at least one processor;
[0066] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any of the above-mentioned human-computer computing task allocation methods based on confidence and cost.
[0067] A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the method for allocating human-computer computing tasks based on confidence and cost described in any one of the above items is implemented.
[0068] Compared with the prior art, the beneficial effects of the present invention are:
[0069] 1. Before assigning tasks, the present invention first analyzes the specific content of the task in detail, and uses advanced algorithms to evaluate the current confidence of humans and machines in the task. Compared with the existing technology, this method greatly improves the accuracy of task assignment and ensures that the most suitable executor completes the task.
[0070] 2. During the task allocation process, the present invention calculates the total expected reward for each task based on the preset task reward indicators and the detailed evaluation of the cost of humans and machines completing the tasks, so that the task allocation decision not only takes cost-effectiveness into consideration, but also dynamically adjusts the allocation strategy to improve the overall resource utilization efficiency.
[0071] 3. The present invention introduces a fusion method for humans and machines to complete tasks together, which reduces the investment in human and machine costs without reducing the task accuracy, and further improves the overall benefits of human-machine collaboration.
[0072] In summary, the present invention performs task allocation by evaluating the confidence of humans and machines and the cost of tasks. First, it determines which party has a higher accuracy rate in completing the task. Then, the task allocation system determines whether the task is completed by humans, machines, or humans and machines together based on the confidence and cost evaluation results. At the same time, a dynamic scheduling algorithm based on real-time data and prediction models is adopted to realize the intelligence and flexibility of task allocation. The present invention can quickly adjust task allocation in a constantly changing environment and improve the accuracy of task completion and resource utilization efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0073] Figure 1 It is a flowchart of the method of the present invention.
[0074] Figure 2 It is a schematic diagram of the human-machine computing task allocation based on confidence and cost of the present invention.
[0075] Figure 3 It is a flow chart of the method for allocating human-computer computing tasks based on confidence and cost of the present invention.
[0076] Figure 4 It is an influence diagram of the accuracy of the task allocation method based on the ResNet-110 model on the CIFAR-10H dataset of the present invention as the threshold changes.
[0077] Figure 5 It is an influence diagram of the accuracy of the task allocation method based on the ResNet-164 model on the CIFAR-10H dataset of the present invention as the threshold changes.
[0078] Figure 6 It is an influence diagram of the accuracy of the task allocation method based on the PreResNet-164 model on the CIFAR-10H dataset of the present invention as the threshold changes.
[0079] Figure 7 It is an influence diagram of the accuracy of the task allocation method based on the DenseNet-BC model on the CIFAR-10H dataset of the present invention as the threshold changes.
[0080] Figure 8 It is an influence diagram of the accuracy of the task allocation method based on the VGG model (noise level 80) on the ImageNet-16H dataset of the present invention as the threshold changes.
[0081] Fig. 9It is an influence diagram of the accuracy of the task allocation method based on the VGG model (noise level 95) on the ImageNet-16H dataset of the present invention as the threshold changes.
[0082] Fig.10 It is an influence diagram of the accuracy of the task allocation method based on the VGG model (noise level 110) on the ImageNet-16H dataset of the present invention as the threshold changes.
[0083] Fig.11 It is an influence diagram of the accuracy of the task allocation method based on the VGG model (noise level 125) on the ImageNet-16H dataset of the present invention as the threshold changes. DETAILED DESCRIPTION
[0084] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and embodiments.
[0085] A confidence- and cost-based method for allocating human-machine computing tasks determines whether the task is completed by humans, machines, or both humans and machines, based on the specific content of the task, while ensuring high accuracy and reducing costs.
[0086] The idea of the present invention is: given the input of a task And the confidence of humans and machines for the current task γ m and γ m , the present invention considers a k-tuple classification problem to predict the label y∈{1, 2, 3, ..., k}, where k represents the number of categories. In addition, the present invention will reward R(x t ), the cost of humans completing the current task C h (x t ), the cost C of the machine completing the current task m (x t ), to calculate the total expected reward for humans and machines to complete the current task
[0087] Task allocation depends on two important indicators: one is the confidence of the human-machine relationship and the other is the total reward for completing the task. The present invention allocates tasks based on these two indicators and determines the decision variable d by the allocation system. h (x t ) and d m (x t ), both of which belong to {0, 1}, where d h (x t ) indicates whether the current task is assigned to humans, d m (x t) indicates whether the current task is assigned to a machine. The final task will be assigned to humans, machines, or humans and machines together. The accuracy results when all tasks are completed by humans and machines together are used as a reference. The goal of this invention is to maximize the task set without reducing the accuracy. The total expected reward can be expressed as:
[0088]
[0089] In order to maximize the total expected reward, when the confidence of both humans and machines is high, it is necessary to calculate the human-machine cost for allocation; when the confidence of both humans and machines is low, the task needs to be assigned to humans and machines together and the results must be integrated.
[0090] The present invention considers image multi-classification tasks and assigns tasks to three types of task executors. The human-machine computing task allocation method based on confidence and cost mainly includes four modules, namely cost calculation, task allocation, result fusion and task executor. First, the task is received, and the task first enters the task allocation. The task allocation determines who is assigned to perform the current task based on the confidence of humans and machines. When the confidence of humans and machines is high, the task enters the class cost calculation module, which calculates the reward of the task and the cost of humans and machines completing the task through the input image, thereby calculating the total expected reward of the task; when the confidence of humans and machines is low, the task will enter the human-machine result fusion cost. The fusion result of humans and machines is always higher than that of humans or machines; the rest of the cases are directly assigned to humans or machines according to the confidence. In the present invention, there are three types of task executors, namely humans, machines, and humans and machines.
[0091] like Figure 1 and Figure 2 As shown, the method includes:
[0092] Step 1, obtain the confidence of the image classification task, the human performer and the machine performer;
[0093] Before assigning tasks, the present invention first conducts a detailed analysis of the specific content of the task and uses advanced algorithms to evaluate the current confidence of humans and machines in the task. Compared with the existing technology, this method greatly improves the accuracy of task assignment and ensures that the most suitable executor completes the task.
[0094] Step 2, setting a confidence threshold, and obtaining two comparison conditions of confidence levels according to the confidence threshold, the confidence of the human executor, and the confidence of the machine executor;
[0095] Step 3: Based on the two comparison conditions of confidence level, the accuracy of the image classification task performed by the human executor and the machine executor is judged, and the executor of the image classification task is determined;
[0096] like Figure 3 As shown, the method for determining the executor of the image classification task is:
[0097] (1) When the confidence of the human performer and the confidence of the machine performer meet one of the comparison conditions, and one of the two performers has a higher accuracy rate in performing the image classification task, the image classification task is assigned to the performer with higher confidence, i.e., the performer with higher accuracy rate;
[0098] (2) When the confidence of the human executor and the confidence of the machine executor both meet the two comparison conditions, and the accuracy of the image classification task performed by the human executor and the machine executor is high, the total expected reward of the human executor and the total expected reward of the machine executor are calculated respectively, and the image classification task is assigned to the party with a higher total expected reward, that is, the party with a higher image classification accuracy;
[0099] During the task allocation process, the present invention calculates the total expected reward for each task based on preset task reward indicators and detailed evaluation of the cost of humans and machines completing the task, so that the task allocation decision not only takes cost-effectiveness into consideration, but also dynamically adjusts the allocation strategy to improve the overall resource utilization efficiency.
[0100] (3) When the confidence of the human performer and the confidence of the machine performer do not meet the two comparison conditions, the accuracy of the image classification task performed by the human performer and the machine performer is low. The image classification task is assigned to the human performer and the machine performer to complete together, and the accuracy of the image classification task is calculated. Specifically: When both conditions are not met, it means that the confidence of humans and machines is low at this time. The accuracy of humans and machines completing the task together is higher than that of humans or machines. At this time, the task is assigned to humans and machines at the same time, and different types of results are obtained. The class-level output of humans and the predicted distribution of machines are fused to obtain the final result. In this case, the task performers are both humans and machines.
[0101] The present invention introduces a fusion method for humans and machines to complete tasks together, which reduces the investment in human and machine costs without reducing the task accuracy, and further improves the overall benefit of human-machine collaboration.
[0102] The confidence of the human executor and the confidence of the machine executor are both obtained from the crowdsourcing platform.
[0103] The two comparison conditions of the confidence level are expressed as:
[0104] η(γ(m,x) ,τ)=w1{|γ (m,x) |>τ}
[0105] η(γ (h,x) ,τ)=w1{| (h,x) |>τ},
[0106] Among them, τ is the confidence threshold, γ h is the confidence of the human performer, γ m is the confidence of the machine executor.
[0107] When the confidence of the human executor and the confidence of the machine executor both meet the two comparison conditions, the accuracy of the image classification task performed by the human executor and the machine executor is high, then the total expected reward of the human executor and the total expected reward of the machine executor are calculated respectively, and the image classification task is assigned to the party with the higher total expected reward, that is, the party with the higher image classification accuracy, specifically:
[0108] For image classification tasks, the man-machine cost is positively correlated with the complexity of the image. In the present invention, two different dimensions of image complexity are considered, namely spatial structure and visual perception complexity.
[0109] Step 2.11, calculate the discrete entropy of the image, taking into account the underlying spatial image structure and pixel co-occurrence through density analysis based on the gradient vectors in the image, expressed as:
[0110]
[0111] Where DE is the discrete entropy of the image, I and J are the number of two-dimensional discrete units of the probability density function, and p i,j is a two-dimensional probability density function, where The factor reflects Papoulis generalized sampling theory, which cuts the entropy rate in half;
[0112] The two-dimensional probability density function is expressed as:
[0113]
[0114] Among them, d x represents the derivative kernel in the x direction, d y represents the derivative kernel in the y direction, δ is the Kronecker delta function that describes the binning process in histogram generation, W is the width of the image, and H is the height of the image;
[0115] Step 2.12, calculate the perimeter complexity of the image, a method to measure the complexity of binary images. This measure was originally called dispersion and is used to describe the perceptual complexity of visual shapes, expressed as:
[0116]
[0117] Among them, PC is the perimeter complexity of the image, P is the perimeter of the image, and A is the area of the image;
[0118] DE and PC are two different dimensions of image complexity. When the image content is messy, without a specific structure or pattern and with low contrast, there will be a higher DE value, and image classification will be more difficult; when there are more textures and edges in the image, and the color changes in the image are complex, with multiple colors and gradients, there will be a higher PC value.
[0119] Step 2.13, use the discrete entropy of the image and the perimeter complexity of the image to calculate the reward after the task is completed, expressed as:
[0120] R(x t )=f1(DE,PC),
[0121] Among them, x t For the current image classification task;
[0122] Step 2.14, calculate the cost of human executors and the cost of machine executors:
[0123] The cost of the human executor is expressed as:
[0124] C h (x t )=f2(DE,PC),
[0125] The cost of the machine executor is expressed as:
[0126] C m (x t )=f3(DE,PC),
[0127] Step 2.15, calculate the total expected reward of the human executor and the total expected reward (profit) of the machine executor based on the reward after the task is completed, the cost of the human executor, and the cost of the machine executor:
[0128] The total expected reward (profit) of the human executor is expressed as:
[0129]
[0130] The total expected reward (profit) of the machine executor is expressed as:
[0131]
[0132] Step 2.16, compare the total expected reward of the human executor and the total expected reward of the machine executor. If the total expected reward of the human executor is greater than the total expected reward of the machine executor, the accuracy of the human executor in performing the image classification task is high, and the image classification task is assigned to the human executor; if the total expected reward of the human executor is less than the total expected reward of the machine executor, the accuracy of the machine executor in performing the image classification task is high, and the image classification task is assigned to the machine executor.
[0133] When the confidence of the human executor and the confidence of the machine executor do not meet the two comparison conditions, the accuracy of the image classification task performed by the human executor and the machine executor is low, the image classification task is assigned to the human executor and the machine executor to complete together, and the accuracy of the image classification task is calculated, specifically:
[0134] The invention is to use both the category-level output provided by humans and the predicted distribution provided by the classifier. The invention focuses on the specific case of humans and machines, and the current setting can be widely applied to the combination of non-probabilistic labelers (whose output is classified) and probabilistic labelers (whose output is the distribution of classes). The invention uses a probabilistic method for modeling, and the conditional distribution of the required label can be decomposed by Bayes' rule as:
[0135] p(y|h(x t ), m(x t ))∝p(h(x t )|y,m(x t ))p(y|m(x t )),
[0136] Among them, h(x t ) is the class-level label predicted by the human performer, m(x t ) is the probability level label predicted by the machine executor, and y is the label of the image classification;
[0137] In this case, assuming that the human executors and machine executors are independent, the conditional distribution probability of each label is calculated under the premise of knowing the class-level label of the human executor and the probability-level prediction of the machine executor:
[0138] p(y|h(x t ), m(x t ))∝p(h(x t )|y)p(y|m(x t )),
[0139] Among them, p(h(x t )|y) is the category-level calibration probability when the human performer performs the image classification task independently, p(y|m(x t)) is the instance-level calibration probability when the machine executor performs the image classification task independently;
[0140] Calibrate the probability-level labels predicted by the machine executor to obtain the calibrated probability-level labels predicted by the machine executor:
[0141] Since the classifier m(x t ) may have the same probability output as p(y|m(x t )). However, modern neural networks tend to be overconfident in their predictions. To address this problem, we transform m(x t ) is calibrated to a good probability a posteriori. Model calibration is performed using MAP temperature scaling (TS). Temperature T∈(0, 1) indicates insufficient confidence and T∈(1, ∞) indicates excessive confidence. To account for the scale difference, we log the temperature A Gaussian prior is imposed. Since this is a non-conjugate prior, the maximum a posteriori (MAP) temperature is estimated via gradient-based optimization.
[0142] Using the probability-level labels predicted by the calibrated machine performer and the confidence of the human performer, the calibrated probability at the category level when the human performer performs the image classification task independently and the calibrated probability at the instance level when the machine performer performs the image classification task independently are fused to obtain the accuracy rate when the human performer and the machine performer complete the image classification task together, which is expressed as:
[0143]
[0144] Among them, m θ (x) is the probability-level label predicted by the calibrated machine performer, T is the temperature, T∈(0,1) indicates insufficient confidence, and T∈(1,∞) indicates too high confidence.
[0145] A human-machine computing task allocation system based on confidence and cost, comprising:
[0146] The information acquisition module obtains the confidence of the image classification task, the human performer, and the machine performer;
[0147] A comparison condition acquisition module sets a confidence threshold and acquires two comparison conditions of confidence levels according to the confidence threshold, the confidence of the human executor, and the confidence of the machine executor;
[0148] The task assignment module determines the accuracy of image classification tasks performed by human and machine performers based on two comparison conditions of confidence level, and decides who will perform the image classification tasks.
[0149] The method for determining the executor of the image classification task is:
[0150] When the confidence of the human executor and the confidence of the machine executor meet one of the comparison conditions, and one of the human executor and the machine executor has a higher accuracy rate in performing the image classification task, the image classification task is assigned to the one with higher confidence, that is, the one with higher accuracy rate in the image classification task;
[0151] When the confidence of the human executor and the confidence of the machine executor both meet the two comparison conditions, and the accuracy of the image classification task performed by the human executor and the machine executor is high, the total expected reward of the human executor and the total expected reward of the machine executor are calculated respectively, and the image classification task is assigned to the party with a higher total expected reward, that is, the party with a higher image classification accuracy;
[0152] When the confidence of the human executor and the confidence of the machine executor do not meet the two comparison conditions, the accuracy of the image classification task performed by the human executor and the machine executor is low. The image classification task is assigned to the human executor and the machine executor to complete together, and the accuracy of the image classification task is calculated.
[0153] An electronic device, comprising:
[0154] at least one processor;
[0155] and, a memory communicatively coupled to the at least one processor;
[0156] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any of the above-mentioned human-computer computing task allocation methods based on confidence and cost.
[0157] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements any of the above methods for allocating human-computer computing tasks based on confidence and cost.
[0158] Figure 4-11 The effect of the human-machine computing task allocation method using confidence and cost on the accuracy of different machine learning models as the threshold changes. Among them, the horizontal axis represents the change of the threshold, the vertical axis is the result of the image classification task prediction, the long dotted line represents the result predicted by the human executor alone, the short dotted line represents the result predicted by the machine executor alone, the blue line is the result of the supervised joint prediction of the human executor and the machine executor, the orange line is the accuracy curve of the result of the unsupervised joint prediction of the human executor and the machine executor, and the red line is the accuracy curve of the prediction result of the method proposed in the present invention. It can be seen from the figure that the present invention is suitable for the optimal allocation of complex tasks, and can maximize the total expected reward of the task without reducing the accuracy.
[0159] In summary, the present invention performs task allocation by evaluating the confidence of humans and machines and the cost of tasks. First, the specific content of the task and the confidence of humans and machines in the current task are evaluated. By evaluating the confidence of humans and machines, it is determined which party has a higher accuracy rate when completing the task. Subsequently, the total expected reward for humans and machines to complete the current task is calculated based on the pre-defined task reward, the cost of humans completing the current task, and the cost of machines completing the current task. On this basis, the task allocation system determines whether the task is completed by humans, machines, or humans and machines together according to the confidence and cost evaluation results. The present invention can not only make full use of the advantages of human-machine collaboration and improve the accuracy of task completion, but also flexibly adjust task allocation in a dynamically changing environment and optimize resource utilization efficiency. By introducing a fusion method of humans and machines completing tasks together, the present invention reduces the investment of human and machine costs without reducing the accuracy of tasks, and further improves the overall benefits of human-machine collaboration.
[0160] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present application, and in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present application.
Claims
1. A method for allocating human-computer computing tasks based on confidence and cost, characterized in that: The method comprises: Step 1, obtain the confidence of the image classification task, the human performer and the machine performer; Step 2, setting a confidence threshold, and obtaining two comparison conditions of confidence levels according to the confidence threshold, the confidence of the human executor, and the confidence of the machine executor; Step 3: Based on the two comparison conditions of confidence level, the accuracy of the image classification task performed by the human executor and the machine executor is judged, and the executor of the image classification task is determined; The method for determining the executor of the image classification task is: When the confidence of the human executor and the confidence of the machine executor meet one of the comparison conditions, and one of the human executor and the machine executor has a higher accuracy rate in performing the image classification task, the image classification task is assigned to the one with higher confidence, that is, the one with higher accuracy rate in the image classification task; When the confidence of the human executor and the confidence of the machine executor both meet the two comparison conditions, and the accuracy of the image classification task performed by the human executor and the machine executor is high, the total expected reward of the human executor and the total expected reward of the machine executor are calculated respectively, and the image classification task is assigned to the party with a higher total expected reward, that is, the party with a higher image classification accuracy; When the confidence of the human executor and the confidence of the machine executor do not meet the two comparison conditions, the accuracy of the image classification task performed by the human executor and the machine executor is low. The image classification task is assigned to the human executor and the machine executor to complete together, and the accuracy of the image classification task is calculated.
2. The method for allocating human-computer computing tasks based on confidence and cost according to claim 1, characterized in that: The confidence of the human executor and the confidence of the machine executor are both obtained from the crowdsourcing platform.
3. The method for allocating human-computer computing tasks based on confidence and cost according to claim 2, characterized in that: The two comparison conditions of the confidence level are expressed as: in, is the confidence threshold, γ h is the confidence of the human performer, γ m is the confidence of the machine executor.
4. The method for allocating human-computer computing tasks based on confidence and cost according to claim 3 is characterized in that: When the confidence of the human executor and the confidence of the machine executor both meet the two comparison conditions, the accuracy of the image classification task performed by the human executor and the machine executor is high, then the total expected reward of the human executor and the total expected reward of the machine executor are calculated respectively, and the image classification task is assigned to the party with the higher total expected reward, that is, the party with the higher image classification accuracy, including: Step 2.11, calculate the discrete entropy of the image, expressed as: Where DE is the discrete entropy of the image, I and J are the number of two-dimensional discrete units of the probability density function, and p i,j is a two-dimensional probability density function; The two-dimensional probability density function is expressed as: Among them, d x represents the derivative kernel in the x direction, d y represents the derivative kernel in the y direction, δ is the Kronecker delta function that describes the binning process in histogram generation, W is the width of the image, and H is the height of the image; Step 2.12, calculate the perimeter complexity of the image, expressed as: Among them, PC is the perimeter complexity of the image, P is the perimeter of the image, and A is the area of the image; Step 2.13, use the discrete entropy of the image and the perimeter complexity of the image to calculate the reward after the task is completed, expressed as: R(x t )=j1(DE,PC), Among them, x t For the current image classification task; Step 2.14, calculate the cost of human executors and the cost of machine executors: The cost of the human executor is expressed as: C h (x t )=f2(DE,PC), The cost of the machine executor is expressed as: C m (x t )=f3(DE,PC), Step 2.15, calculate the total expected reward of the human executor and the total expected reward of the machine executor based on the reward after the task is completed, the cost of the human executor, and the cost of the machine executor: The total expected reward for the human performer is expressed as: The total expected reward of the machine executor is expressed as: Step 2.16, compare the total expected reward of the human executor and the total expected reward of the machine executor. If the total expected reward of the human executor is greater than the total expected reward of the machine executor, the accuracy of the human executor in performing the image classification task is high, and the image classification task is assigned to the human executor; if the total expected reward of the human executor is less than the total expected reward of the machine executor, the accuracy of the machine executor in performing the image classification task is high, and the image classification task is assigned to the machine executor.
5. The method for allocating human-computer computing tasks based on confidence and cost according to claim 4, characterized in that: When the confidence of the human executor and the confidence of the machine executor do not meet the two comparison conditions, the accuracy of the image classification task performed by the human executor and the machine executor is low, the image classification task is assigned to the human executor and the machine executor to complete together, and the accuracy of the image classification task is calculated, specifically: Step 2.21, label different categories in the image classification task; Step 2.22, use the Bayesian rule to get the conditional probability of each classification label, expressed as: p(y|h(x t ),m(x t ))∝p(h(x t )|y,m(x t ))p(y|m(x t )), Among them, h(x t ) is the class-level label predicted by the human performer, m(x t ) is the probability level label predicted by the machine executor, and y is the label of the image classification; Step 2.23, when both the human performer and the machine performer complete the image classification task independently, the conditional distribution probability of each label is expressed as: p(y|h(x t ),m(x t ))∝p(h(x t )|y)p(y|m(x t )), Among them, p(h(x t )|y) is the category-level calibration probability when the human performer performs the image classification task independently, p(y|m(x t )) is the instance-level calibration probability when the machine executor performs the image classification task independently; Step 2.24, calibrating the probability level labels predicted by the machine executor to obtain the calibrated probability level labels predicted by the machine executor; Step 2.25, using the probability-level labels predicted by the calibrated machine performer and the confidence of the human performer, the calibrated probability of the category level when the human performer performs the image classification task independently and the calibrated probability of the instance level when the machine performer performs the image classification task independently are fused to obtain the accuracy of the image classification task completed by the human performer and the machine performer together, expressed as: Where m(x) is the probability-level label predicted by the calibrated machine performer, T is the temperature, T∈(0,1) indicates insufficient confidence, and T∈(1,∞) indicates too high confidence.
6. A human-computer computing task allocation system based on confidence and cost, characterized in that: include: The information acquisition module obtains the confidence of the image classification task, the human performer, and the machine performer; A comparison condition acquisition module sets a confidence threshold and acquires two comparison conditions of confidence levels according to the confidence threshold, the confidence of the human executor, and the confidence of the machine executor; The task assignment module determines the accuracy of image classification tasks performed by human and machine performers based on two comparison conditions of confidence level, and decides who will perform the image classification tasks. The method for determining the executor of the image classification task is: When the confidence of the human executor and the confidence of the machine executor meet one of the comparison conditions, and one of the human executor and the machine executor has a higher accuracy rate in performing the image classification task, the image classification task is assigned to the one with higher confidence, that is, the one with higher accuracy rate in the image classification task; When the confidence of the human executor and the confidence of the machine executor both meet the two comparison conditions, and the accuracy of the image classification task performed by the human executor and the machine executor is high, the total expected reward of the human executor and the total expected reward of the machine executor are calculated respectively, and the image classification task is assigned to the party with a higher total expected reward, that is, the party with a higher image classification accuracy; When the confidence of the human executor and the confidence of the machine executor do not meet the two comparison conditions, the accuracy of the image classification task performed by the human executor and the machine executor is low. The image classification task is assigned to the human executor and the machine executor to complete together, and the accuracy of the image classification task is calculated.
7. An electronic device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; In which, the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a human-computer computing task allocation method based on confidence and cost as described in any one of claims 1 to 5.
8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, a method for allocating human-computer computing tasks based on confidence and cost as described in any one of claims 1 to 5 is implemented.
