A data labeling crowdsourcing method and system based on blockchain and crowd wisdom

Abstract

The application discloses a data labeling crowd sourcing method and system based on block chain and crowd wisdom, comprising the steps of publishing a labeling task and locking a reward; a trusted execution environment (TEE) node and a labeler are registered and funds are pledged respectively; a labeler cluster is selected based on multi-dimensional ability modeling; a specified number of nodes are selected from the TEE node by a random selection algorithm to form an executed TEE cluster; the labeling result of the labeler cluster is encrypted and sent to the selected TEE cluster; the TEE node executes a predefined aggregation algorithm and benefit calculation in a secure environment, and reports the result; if it is invalid, a backup node is started to replace the execution; the execution results are compared, if they are consistent, the benefits are distributed; if they are inconsistent, execution is performed until a consistent result is obtained. The application selects the labeler cluster with the most crowd wisdom, and makes the labeling result of the labeler cluster pass through the redundant aggregation and clearing of the multi-TEE cluster selected at random, so that the crowd labeling with high quality and high credibility is realized.

Description

Technical Field

[0001] This invention relates to the field of data annotation technology, and in particular to a data annotation crowdsourcing method and system based on blockchain and collective intelligence. Background Technology

[0002] With the increasingly widespread application of artificial intelligence technology, high-quality data annotation has become a bottleneck for model training. Supervised learning models require a large amount of labeled data before training, and these labels often need to be manually annotated. Traditional manual annotation methods are costly, and the problem of low data annotation quality urgently needs to be addressed. Crowdsourcing platforms have gradually become an effective alternative to improve annotation speed and reduce costs, but this has brought new problems: the varying skills of annotators lead to inconsistent annotation quality, and platforms and task publishers often do not promptly disclose the correctness of each labeled sample, making annotators' rights vulnerable to infringement and increasing the difficulty of data quality control.

[0003] Traditional quality control methods include using gold standard datasets to screen for high-quality annotators. However, this approach not only increases additional annotation costs but also incurs high costs for selecting high-quality annotators. Swarm intelligence theory offers a new approach to solving hierarchical annotation quality problems—improving the accuracy of overall judgment through the aggregation of individual diversity. However, this theory remains theoretical and has not yet been widely applied in practical scenarios, especially lacking concrete technical implementation methods and reliable technical safeguards.

[0004] Trusted Execution Environment (TEE) technology provides a secure computing environment that ensures the isolated execution of data and operations, thereby guaranteeing the security of the computing process and the reliability of the results. Despite its strong security performance, TEE technology still faces a number of challenges in practical applications, particularly side-channel attacks at the physical layer and vulnerability attacks at the software layer. These security vulnerabilities limit the widespread application of TEE technology in certain high-security scenarios, necessitating the design of more secure and reliable technical solutions to meet practical needs.

[0005] In summary, existing technologies have significant shortcomings in terms of cost and quality control in crowdsourcing labeling processes. While the theory of collective intelligence and TEE technology can theoretically solve some problems, they still lack effective implementation methods and safeguards in practical applications. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the existing technology. To achieve the above objectives, a data labeling crowdsourcing method and system based on blockchain and collective intelligence is adopted to solve the problems mentioned in the background technology.

[0007] A data labeling crowdsourcing method based on blockchain and collective intelligence includes the following steps:

[0008] S1. Task Release and Preparation Steps: The data requester releases the annotation task and locks the reward; the Trusted Execution Environment (TEE) node and the annotator register and pledge funds respectively.

[0009] S2. Group building steps: Based on multi-dimensional capability modeling, select the group of annotators who meet the capability threshold and have the greatest potential for collective wisdom from the registered annotators;

[0010] S3. Node selection step: Select a specified number of nodes from the registered TEE nodes through a random selection algorithm to form a TEE cluster to perform this round of aggregation and settlement tasks;

[0011] S4, Trusted Execution and Candidate Steps: The annotation results of the annotator cluster are encrypted and sent to the TEE cluster selected in S3; the TEE nodes execute the predefined aggregation algorithm and profit calculation in a secure environment and report the results; if the execution node fails, a candidate node is activated to take over the execution.

[0012] S5. Decision and Settlement Steps: Compare the execution results of all TEE nodes. If they are consistent, the final profit distribution will be made based on the result. If they are inconsistent, a new execution round will be triggered until a consistent result is obtained.

[0013] As a further aspect of the present invention, step S2 specifically includes the following steps:

[0014] S21. The task issuer extracts a portion of the gold standard dataset with known real label information as the test set, sets an accuracy threshold on the test set, and allows registered annotators to perform test annotations first; preprocesses the image instances in the test set and extracts high-dimensional embedding features.

[0015] S22. Perform dimensionality reduction and standardization on the high-dimensional embedding features to obtain the low-dimensional feature vector x of the instance;

[0016] S23. For users whose annotation results reach a set accuracy threshold, a probabilistic graphical model is constructed based on the annotation result L and the low-dimensional feature vector x, combined with signal detection theory. Through alternating optimization using maximum a posteriori estimation and gradient ascent, the optimized annotator capability weight vector is obtained. ;

[0017] S24. Based on the annotator capability weight vector Calculate the group diversity index D, and use a greedy algorithm to select the annotator group that maximizes the diversity index D; the annotator group annotates the dataset to be annotated in the task, and collects the annotation results.

[0018] As a further aspect of the present invention: the signal detection theory in step S23 assumes that the labeler's label decision L is based on whether the signal U exceeds a decision threshold τ, wherein the signal U is determined by a capability weight vector. It is obtained by performing an inner product with the eigenvector x containing noise n.

[0019] As a further aspect of the present invention: the diversity index D in step S24 is calculated by the sum of the covariances of the ability weight vector representing the labeler and the mean ability vector of the group.

[0020] As a further aspect of the present invention: the random selection algorithm in the node filtering step of step S3 is as follows:

[0021] Define a function that receives the block hash value seed, the total number of TEE nodes M, and the number of nodes to be selected ks;

[0022] By iteratively calculating hash values ​​and performing modulo operations, unique node numbers are added to the set until ks nodes are selected.

[0023] As a further aspect of the present invention: in the trusted execution and candidate steps of step S4, the annotation result is encrypted using the public key of the selected TEE node and forwarded by the relay contract; after the TEE node decrypts and performs calculations in a secure environment, it sends the result and digital signature to the verification contract.

[0024] For each sample j, the following data is obtained:

[0025] Multiple annotators provided labels ;

[0026] Each annotator has a capability weight vector. ;

[0027] For each candidate label y, calculate its weighted score. :

[0028]

[0029] The final output labels are:

[0030]

[0031] Where j is the index of the sample to be labeled; i is the index of the labeler; The annotation result for sample j by annotator i; Let i be the capability weight vector of the annotator. Let be the magnitude of the capability weight vector of annotator i; Let j be the candidate label for sample j; The score obtained for the candidate label y of sample j; Let j be the final label for sample j; The label with the highest score among all candidate labels for sample j.

[0032] As a further aspect of the present invention: in the trusted execution and backup step of step S4, when it is detected that the execution node stops service or fails to report results after timeout, the same number of nodes are automatically randomly selected from the backup node pool to take over the task.

[0033] As a further aspect of the present invention: for TEE nodes that fail to reconnect successfully or fail to send results on time, a portion of their pledged amount is deducted proportionally, and the task publisher is not required to pay the computing service fee for that node.

[0034] As a further aspect of the present invention: in step S5, the adjudication and settlement step, if the execution results of the TEE nodes are inconsistent, the process re-enters step S3, the node screening step, and step S4, the trusted execution and candidate step, until a consistent calculation result is obtained.

[0035] The second aspect of the technical solution: A labeling system employing a data labeling crowdsourcing method based on blockchain and collective intelligence as described in any of the above embodiments, comprising:

[0036] The group building module is used to select the annotator cluster with the greatest potential for collective intelligence from the registered annotators based on multi-dimensional capability modeling.

[0037] The task publishing and registration module is used for data annotation requesters to publish annotation requests and lock in rewards, as well as for TEE nodes and annotators to register and pledge.

[0038] The node selection module is used to select a cluster of TEE nodes from the registered TEE nodes to perform the current round of aggregation and settlement tasks using a random selection algorithm.

[0039] The data relay module is used to receive encrypted annotation results and forward them to the selected TEE nodes;

[0040] The Trusted Execution Module (TEE) is deployed on TEE nodes and is used to decrypt annotation results, execute aggregation algorithms, and calculate profits in a secure environment.

[0041] The verification and adjudication module is used to receive and compare the execution results of all TEE nodes, trigger the candidate mechanism, and adjudicate the final valid result.

[0042] The clearing and settlement module is used to automatically distribute revenue to annotators and TEE nodes based on the final consistent aggregated clearing results.

[0043] Compared with the prior art, the present invention has the following technical advantages:

[0044] The above technical solution involves two key stages: First, based on multi-dimensional capability modeling and group diversity indicators, a high-quality annotator cluster with the greatest potential for "collective intelligence" is intelligently selected from numerous applicants to ensure annotation quality from the source. Then, the results of these annotators are handed over to an off-chain TEE node cluster selected by a random algorithm for redundant computation. Through encrypted transmission, secure aggregation and settlement, result comparison, and a backup mechanism, the immutability, verifiability, and high fault tolerance of the entire computation process are ensured, and settlement is only triggered after consensus is reached.

[0045] By using "collective intelligence" to screen, the accuracy and reliability of the final aggregated labeling results are effectively improved without significantly increasing the cost of top-tier human resources. At the same time, by combining blockchain with redundant execution of multiple TEEs, a decentralized and trustless execution environment is constructed, which fundamentally prevents the possibility of the platform or task party committing malicious acts in the aggregation and settlement process, protects the labor income of labelers, and achieves a two-way improvement in labeling quality and process credibility. Attached Figure Description

[0046] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings:

[0047] Figure 1 This is a schematic diagram illustrating the steps of the data annotation crowdsourcing method according to an embodiment of this application;

[0048] Figure 2 This is a flowchart illustrating a data annotation crowdsourcing method according to an embodiment of this application. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] In this embodiment, the data annotation crowdsourcing method is applied to the crowdsourcing annotation scenario. Without increasing the annotation cost and the cost of excellent annotation manpower, it enhances the final aggregate annotation effect based on the theory of collective intelligence, and ensures the authenticity and reliability of the settlement result through off-chain multi-TEE redundant execution, thereby achieving a win-win situation for both the data requester and the annotator.

[0051] The main steps involved in execution are as follows:

[0052] The purpose of building a swarm intelligence effect based on multidimensional capability modeling is to select clusters with the greatest potential for collective intelligence from among the registered annotators who meet certain capability thresholds.

[0053] The aggregation and revenue settlement are performed redundantly across multiple TEEs. The annotation results of the annotators are uploaded to a randomly selected off-chain TEE cluster to perform the aggregation of annotation results and revenue calculation.

[0054] Please refer to Figure 1 and Figure 2 In this embodiment of the invention, a data labeling crowdsourcing method based on blockchain and collective intelligence includes the following steps:

[0055] S1. Task Release and Preparation Steps: The data requester releases the annotation task and locks the reward; the Trusted Execution Environment (TEE) node and the annotator register and pledge funds respectively.

[0056] In this embodiment, the preparation stage is as follows:

[0057] The data annotation requester publishes an annotation request [task] contract, including the task dataset, task description, annotation requirements, and reward mechanism. The annotation reward is locked in advance in the [settlement] contract. The annotation task dataset is stored in an encrypted storage location to reduce on-chain storage burden and ensure privacy (IPFS). TEE nodes can register and authenticate their identity by staking a certain amount of deposit in the contract and submit their corresponding public key information to facilitate subsequent remote authentication confirmation. Annotators decide whether to participate in the annotation task based on whether their capabilities meet the annotation requirements and the overall reward. Annotators who ultimately participate in the task need to stake an additional amount of funds to prevent malicious behavior.

[0058] S2. Group Construction Steps: Based on multi-dimensional capability modeling, a group of annotators who meet the capability threshold and have the greatest potential for collective intelligence is selected from the registered annotators. The specific steps include:

[0059] S21. The task issuer extracts a portion of the gold standard dataset with known real label information as the test set, sets an accuracy threshold on the test set, and allows registered annotators to perform test annotations first; preprocesses the image instances in the test set and extracts high-dimensional embedding features.

[0060] Extract the feature vector x of the data instance, i.e., preprocessing;

[0061] For image instances in the preprocessed dataset, they are first processed by a pre-trained convolutional neural network or other model to extract the high-dimensional embedding of the image.

[0062] S22. Perform dimensionality reduction and standardization on the high-dimensional embedding features to obtain the low-dimensional feature vector x of the instance;

[0063] Then, dimensionality reduction techniques such as PCA and t-SNE are used to extract low-dimensional representations of key attributes in the image from high-dimensional features, and the dimensionality-reduced data is standardized.

[0064] Among them, the high-dimensional embedding extracted by the convolutional neural network (CNN) contains feature vectors with approximately thousands of dimensions, while the low-dimensional representation feature vectors processed by dimensionality reduction techniques contain only a few key dimensions such as color and shape, which are used to represent important features that humans can perceive when making judgments; data standardization operations include taking the zero mean, unit variance, etc.

[0065] S23. For users whose annotation results reach a set accuracy threshold, a probabilistic graphical model is constructed based on the annotation result L and the low-dimensional feature vector x, combined with signal detection theory. Through alternating optimization using maximum a posteriori estimation and gradient ascent, the optimized annotator capability weight vector is obtained. ;

[0066] Then, the multidimensional feature capability modeling data is calculated:

[0067] Based on the annotation results L of the above instances and the obtained low-dimensional feature vector x of the instances, combined with signal detection theory, the annotation results are modeled as a probabilistic graphical model, resulting in the joint probability distribution p(L,x, The log-likelihood function m(x, τ) The log-likelihood function with optimal parameters (τ) is found through maximum a posteriori (MAP) estimation. Finally, the log-likelihood function is maximized using an alternating gradient ascent optimization method until convergence, thus obtaining the optimized annotator capability weight vector. .

[0068] S24. Based on the annotator capability weight vector Calculate the group diversity index D, and use a greedy algorithm to select the annotator group that maximizes the diversity index D; the annotator group annotates the dataset to be annotated in the task, and collects the annotation results.

[0069] In this embodiment, the signal detection theory in step S23 assumes that the labeler's label decision L is based on whether the signal U exceeds the decision threshold τ, wherein the signal U is determined by the capability weight vector. It is obtained by performing an inner product with the eigenvector x containing noise n.

[0070] In this context, the signal detection theory assumes that the labeler's label decision L is based on whether the signal U exceeds a decision threshold τ, and the signal U is determined by the capability weight vector. The result is obtained by performing an inner product with the feature vector x containing noise n; the probabilistic graphical model uses nodes to represent observable and latent variables and edges to reflect the conditional dependencies in the probability distribution, thereby constructing an inference framework that integrates observable and latent variables.

[0071] Maximum a posteriori (MAP) estimation makes full use of prior knowledge and likelihood, providing a more accurate (x, ,τ) parameter estimation; the specific process of the gradient ascent alternating optimization method includes alternating fixing of the log-likelihood function m(x, The image parameters x and annotator parameters (τ) in the image parameter x and annotator parameters (τ) ,τ), and perform gradient ascent optimization on another pair of parameters until convergence.

[0072] In this embodiment, the diversity index D in step S24 is calculated by the sum of the covariance of the ability weight vector representing the labeler and the mean ability vector of the group.

[0073] Finally, individual selection is based on the dimension of collective intelligence potential;

[0074] We design a diversity index D for the group and use it as an indicator to measure the collective intelligence. We then use a greedy algorithm to select the group of annotators who meet the initial set number of participants and have the highest diversity index from among the registered participants.

[0075] Among them, the diversity index D of the group is used to measure the potential of collective intelligence contained in the annotation cluster. It is calculated by the sum of the covariance of the ability weight vector representing the annotator and the mean ability vector of the group. The core strategy of the greedy algorithm is to select an annotator in each iteration to maximize the ability diversity of the current user subset S, and then remove the individual from the available annotation group. This process is repeated until a specified number of annotators are selected.

[0076] This completes the screening and construction of a high-quality labeled population with collective intelligence potential.

[0077] S3. Node selection step: Select a specified number of nodes from the registered TEE nodes through a random selection algorithm to form a TEE cluster to perform this round of aggregation and settlement tasks;

[0078] In this embodiment, the random selection algorithm in the node filtering step of step S3 is as follows:

[0079] Define a function that receives the block hash value seed, the total number of TEE nodes M, and the number of nodes to be selected ks;

[0080] By iteratively calculating hash values ​​and performing modulo operations, unique node numbers are added to the set until ks nodes are selected.

[0081] Specifically, the TEE service node selection phase:

[0082] A function select_random_tee_nodes() is defined to randomly select multiple nodes to execute TEE services. It takes three parameters: the latest block hash value seed, the current total number of TEE nodes M, and the number of nodes to be selected ks.

[0083] In the `select_random_tee_nodes()` function of the `selecting` contract, a loop is used to find the TEE nodes that participate in each round of aggregation and liquidation.

[0084] Calculate seed_int = INTERGER(h(seed + r));

[0085] Where seed_int is the block hash value, and r is the counter;

[0086] Perform a modulo operation on seed_int with M to obtain the number i;

[0087] If the obtained mapping number i is not in the set, then add it to the initially empty set set;

[0088] If the number is duplicated in the set, then the addition is abandoned;

[0089] Calculate seed = h(seed) and increment the counter r to update the random number seed and counter. Proceed to the next round of the loop until the set contains the number of nodes equal to ks, at which point the loop ends.

[0090] The function h accepts the parameter "seed + r", converts it into a string and then into a SHA-256 hash value; the MOD operation converts the input SHA-256 hash value into an integer number form, obtaining a TEE node number mapping in the range [0, M-1].

[0091] The algorithm described above can generate a random set of TEE nodes from a TEE cluster to participate in the current round of aggregation and liquidation. Unselected TEE nodes can then be added to a candidate pool to await opportunities for subsequent task execution. Finally, the [selecting] contract records the addresses and other identity information of these nodes.

[0092] S4, Trusted Execution and Candidate Steps: The annotation results of the annotator cluster are encrypted and sent to the TEE cluster selected in S3; the TEE nodes execute the predefined aggregation algorithm and profit calculation in a secure environment and report the results; if the execution node fails, a candidate node is activated to take over the execution.

[0093] In this embodiment, in the trusted execution and candidate steps of step S4, the annotation result is encrypted using the public key of the selected TEE node and forwarded by the relay contract; after the TEE node decrypts and performs calculations in a secure environment, it sends the result and digital signature to the verification contract.

[0094] For each sample j, the following data is obtained:

[0095] Multiple annotators provided labels ;

[0096] Each annotator has a capability weight vector. ;

[0097] For each candidate label y, calculate its weighted score. :

[0098]

[0099] The final output labels are:

[0100]

[0101] Where j is the index of the sample to be labeled; i is the index of the labeler; The annotation result for sample j by annotator i; Let i be the capability weight vector of the annotator. Let be the magnitude of the capability weight vector of annotator i; Let j be the candidate label for sample j; The score obtained for the candidate label y of sample j (the formula means: all samples j labeled as...) (The sum of the modulo of the annotators); Let j be the final label for sample j; The label with the highest score among all candidate labels for sample j.

[0102] In this embodiment, in the trusted execution and backup step S4, when it is detected that the execution node stops service or fails to report results after timeout, the same number of nodes are automatically randomly selected from the backup node pool to take over the task.

[0103] In this embodiment, for TEE nodes that fail to reconnect successfully or fail to send results on time, a portion of their pledged amount is deducted proportionally, and the task publisher does not need to pay the computing service fee for that node.

[0104] In this embodiment, the execution phase is as follows:

[0105] The selected TEE node updates its local execution environment and then sends remote authentication and signature information to the [selecting] contract to ensure that the executed software and hardware environment are consistent with the requirements and that the identity information is compliant.

[0106] Once all annotators have completed their annotation tasks, they encrypt their annotation results in batches using the public key set of the aforementioned TEE nodes and send the encrypted results to the [relaying] contract, which then forwards them to the corresponding TEE nodes. The TEE nodes decrypt the annotation results using their private keys, execute the predefined aggregation algorithm and revenue settlement mechanism, and wait for all TEE nodes to complete the aggregation and settlement tasks and send their execution results and corresponding digital signatures to the [validation] contract.

[0107] In this embodiment, the waiting stage:

[0108] To improve the stability of the entire system and prevent TEE nodes from going offline or being attacked during the execution of computing tasks, a backup phase is set up: when a node in a computing task stops serving or fails to send the computing result to the [validation] contract within a timeout period, the [validation] contract will automatically trigger the backup function, randomly selecting the same number of TEE nodes from the candidate nodes registered at the time of registration using the above-mentioned random selection algorithm, and the [relaying] contract will send the encrypted annotation result to them to perform the same aggregation and liquidation task.

[0109] For nodes that are temporarily offline, as long as they successfully reconnect within the specified time and upload the results on time, it will not have a significant impact on the system; only a mild penalty will be imposed.

[0110] For nodes that fail to reconnect successfully or fail to send results on time, a portion of their pledged amount will be deducted proportionally, and the task publisher will not be required to pay the computing service fee for that node.

[0111] S5. Decision and Settlement Steps: Compare the execution results of all TEE nodes. If they are consistent, the final profit distribution will be made based on the result. If they are inconsistent, a new execution round will be triggered until a consistent result is obtained.

[0112] In this embodiment, if the execution results of the TEE nodes are inconsistent in step S5 (adjudication and settlement), the process will re-enter step S3 (node ​​screening) and step S4 (trusted execution and candidate) until a consistent calculation result is obtained.

[0113] Specifically, the adjudication and settlement phase:

[0114] During the settlement phase, the execution results of all TEE nodes have been sent to the [validation] contract, which will automatically compare these results:

[0115] If all results are consistent, they are transmitted as trusted results to the [settlement] contract, which will then automatically distribute the revenue to the annotators and TEE nodes according to the results.

[0116] If inconsistent results occur, the process re-enters the waiting phase to select a new round of execution nodes until all TEE nodes produce consistent calculation results.

[0117] Based on the final and consistent aggregated settlement results, the correct or incorrect execution of historical tasks by each node will be determined. For incorrect execution results involving fraudulent behavior, a portion of the deposit paid during registration may be deducted or the node may be prohibited from participating in subsequent computational tasks, depending on the circumstances.

[0118] Simultaneous system security and reliability analysis:

[0119] From a rational perspective, there is no direct zero-sum interest relationship between honest TEE nodes and publishers and annotators. At the same time, from their own perspective, in order to earn the reward for computing tasks and prevent their deposits from being deducted, there is no reason for them to arbitrarily abandon the execution of tasks or maliciously forge aggregation and settlement results. Therefore, it is reasonable to assume that honest nodes will eventually provide correct and timely aggregation and settlement results.

[0120] When there are only a few malicious nodes in the system, their aggregated liquidation results are inconsistent with those of honest nodes. At this time, a new set of TEE nodes will be randomly selected to execute the task until the correct and consistent execution result is obtained. The previously given incorrect results will also be identified and punished accordingly.

[0121] Since the TEE nodes participating in the task are randomly selected, it is difficult for them to predict the identities of all other TEE participants in advance and to conspire in advance. Therefore, it is impossible for all the randomly selected nodes to be malicious nodes, set up the exact same backdoors, and give false consistency results to successfully commit malicious acts.

[0122] The second aspect of the technical solution: A labeling system employing a data labeling crowdsourcing method based on blockchain and collective intelligence as described in any of the above embodiments, comprising:

[0123] The group building module is used to select the annotator cluster with the greatest potential for collective intelligence from the registered annotators based on multi-dimensional capability modeling.

[0124] The task publishing and registration module is used for data annotation requesters to publish annotation requests and lock in rewards, as well as for TEE nodes and annotators to register and pledge.

[0125] The node selection module is used to select a cluster of TEE nodes from the registered TEE nodes to perform the current round of aggregation and settlement tasks using a random selection algorithm.

[0126] The data relay module is used to receive encrypted annotation results and forward them to the selected TEE nodes;

[0127] The Trusted Execution Module (TEE) is deployed on TEE nodes and is used to decrypt annotation results, execute aggregation algorithms, and calculate profits in a secure environment.

[0128] The verification and adjudication module is used to receive and compare the execution results of all TEE nodes, trigger the candidate mechanism, and adjudicate the final valid result.

[0129] The clearing and settlement module is used to automatically distribute revenue to annotators and TEE nodes based on the final consistent aggregated clearing results.

[0130] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention. The scope of the invention is defined by the appended claims and their equivalents, all of which should be included within the scope of protection of the invention.

Claims

1. A data labeling crowdsourcing method based on blockchain and collective intelligence, characterized in that, Includes the following steps: S1. Task Release and Preparation Steps: The data requester releases the annotation task and locks the reward; the Trusted Execution Environment (TEE) node and the annotator register and pledge funds respectively. S2. Group Construction Steps: Based on multi-dimensional capability modeling, select the group of annotators who meet the capability threshold and have the greatest potential for collective intelligence from the registered annotators, and collect the annotation results of this annotator group. The specific steps include: S21. The task issuer extracts a portion of the gold standard dataset with known real label information as the test set, sets an accuracy threshold on the test set, and allows registered annotators to perform test annotations first; preprocesses the image instances in the test set and extracts high-dimensional embedding features. S22. Perform dimensionality reduction and standardization on the high-dimensional embedding features to obtain the low-dimensional feature vector x of the instance; S23. For users whose annotation results reach a set accuracy threshold, a probabilistic graphical model is constructed based on the annotation result L and the low-dimensional feature vector x, combined with signal detection theory. Through alternating optimization using maximum a posteriori estimation and gradient ascent, the optimized annotator capability weight vector is obtained. ; S24. Based on the annotator capability weight vector Calculate the population diversity index D, and use a greedy algorithm to select the annotator group that maximizes the diversity index D; the annotator group annotates the dataset to be annotated in the task, and collects the annotation results; S3. Node selection step: Select a specified number of nodes from the registered TEE nodes through a random selection algorithm to form a TEE cluster to perform this round of aggregation and settlement tasks; S4, Trusted Execution and Candidate Steps: The annotation results of the annotator cluster are encrypted and sent to the TEE cluster selected in S3; the TEE nodes execute the predefined aggregation algorithm and profit calculation in a secure environment and report the results; if the execution node fails, a candidate node is activated to take over the execution. S5. Decision and Settlement Steps: Compare the execution results of all TEE nodes. If they are consistent, the final profit distribution will be made based on the result. If they are inconsistent, a new execution round will be triggered until a consistent result is obtained.

2. The data labeling crowdsourcing method based on blockchain and collective intelligence according to claim 1, characterized in that, The signal detection theory in step S23 assumes that the labeler's label decision L is based on whether the signal U exceeds the decision threshold τ, where the signal U is determined by the capability weight vector. It is obtained by performing an inner product with the eigenvector x containing noise n.

3. The data labeling crowdsourcing method based on blockchain and collective intelligence according to claim 1, characterized in that, The diversity index D in step S24 is calculated by the sum of the covariances of the ability weight vector representing the labeler and the mean ability vector of the group.

4. The data labeling crowdsourcing method based on blockchain and collective intelligence according to claim 1, characterized in that, The random selection algorithm in the node filtering step of step S3 is as follows: Define a function that receives the block hash value seed, the total number of TEE nodes M, and the number of nodes to be selected ks; By iteratively calculating hash values ​​and performing modulo operations, unique node numbers are added to the set until ks nodes are selected.

5. The data labeling crowdsourcing method based on blockchain and collective intelligence according to claim 1, characterized in that, In the trusted execution and candidate steps of step S4, the annotation result is encrypted using the public key of the selected TEE node and forwarded by the relay contract; after the TEE node decrypts and performs the calculation in a secure environment, it sends the result and digital signature to the verification contract. For each sample j, the following data is obtained: Multiple annotators provided labels ; Each annotator has a capability weight vector. ; For each candidate label y, calculate its weighted score. : The final output labels are: Where j is the index of the sample to be labeled; i is the index of the labeler; The annotation result for sample j by annotator i; Let i be the capability weight vector of the annotator. Let be the magnitude of the capability weight vector of annotator i; Let j be the candidate label for sample j; The score obtained for the candidate label y of sample j; Let j be the final label for sample j; The label with the highest score among all candidate labels for sample j.

6. The data labeling crowdsourcing method based on blockchain and collective intelligence according to claim 5, characterized in that, In the trusted execution and backup step of step S4, when it is detected that the execution node stops service or fails to report results after timeout, the same number of nodes are automatically randomly selected from the backup node pool to take over the task.

7. The data labeling crowdsourcing method based on blockchain and collective intelligence according to claim 6, characterized in that, For TEE nodes that fail to reconnect successfully or fail to send results on time, a portion of their pledged amount will be deducted proportionally, and the task publisher will not be required to pay the computing service fee for that node.

8. The data labeling crowdsourcing method based on blockchain and collective intelligence according to claim 1, characterized in that, If the execution results of the TEE nodes are inconsistent in step S5 (adjudication and settlement), the process will re-enter step S3 (node ​​screening) and step S4 (trusted execution and candidate) until a consistent calculation result is obtained.

9. A data annotation crowdsourcing system employing the data annotation method based on blockchain and collective intelligence as described in any one of claims 1 to 8, characterized in that, include: The group building module is used to select the annotator cluster with the greatest potential for collective intelligence from the registered annotators based on multi-dimensional capability modeling. The task publishing and registration module is used for data annotation requesters to publish annotation requests and lock in rewards, as well as for TEE nodes and annotators to register and pledge. The node selection module is used to select a cluster of TEE nodes from the registered TEE nodes to perform the current round of aggregation and settlement tasks using a random selection algorithm. The data relay module is used to receive encrypted annotation results and forward them to the selected TEE nodes; The Trusted Execution Module (TEE) is deployed on TEE nodes and is used to decrypt annotation results, execute aggregation algorithms, and calculate profits in a secure environment. The verification and adjudication module is used to receive and compare the execution results of all TEE nodes, trigger the candidate mechanism, and adjudicate the final valid result. The clearing and settlement module is used to automatically distribute revenue to annotators and TEE nodes based on the final consistent aggregated clearing results.

Images