Data annotation management method and system based on block chain technology

Abstract

The embodiment of the invention discloses a block chain technology-based data annotation management method and system. The method comprises the steps of submitting a to-be-annotated data source and a data annotation demand corresponding to the to-be-annotated data source through a data annotation demand side node; downloading a data annotation work order block through the platform operation node, decrypting the data annotation work order block, encrypting work order assignment data based on a fourth encryption key of a fourth key pair, and uploading the encrypted work order assignment data and a fourth decryption key to a work order assignment block; downloading a work order assignment block through a data labeling operation party node, and performing encryption processing on each data source and a data labeling result corresponding to the data source based on a fifth encryption key of a fifth key pair; performing encryption processing on the data annotation result file based on a sixth encryption key of a sixth key pair; and performing income settlement on each data labeling person corresponding to the work order based on the acceptance report. The problems that in the prior art, data safety in the labeling process cannot be guaranteed, and data labeling services are not fair and rigorous enough are solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a data annotation management system, method, electronic device and storage medium based on blockchain technology. Background Art

[0002] Data labeling work orders refer to tasks or requests to process data such as images, voice, text, and video, mark the features of objects, and generate readable data codes that meet the requirements of machine learning training. This process involves converting raw data into a form that can be understood by machine learning algorithms. Through the labeled data, machines can learn various tasks such as classification, regression, and target detection. The current data labeling market is huge, but data labeling businesses are subject to data leakage, disputes over data labeling results, unfair data labeling quality judgments, and serious data labeling arrears.

[0003] Therefore, there is an urgent need for a method that can ensure the data security of the labeling process and can fairly and rigorously implement initiation, assignment, acceptance, result evaluation and payment settlement. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a data labeling management system, method, electronic device and storage medium based on blockchain technology, so as to solve the problems in the prior art that the data security of the labeling process cannot be guaranteed and the data labeling business is not fair and rigorous enough.

[0005] To achieve the above purpose, an embodiment of the present invention provides a data annotation management method based on blockchain technology, and the method specifically includes:

[0006] Constructing a consortium chain system, wherein the consortium chain roles on the consortium chain system include platform operation nodes, data annotation demanders, and data annotation operators;

[0007] The data security rules built based on the alliance chain system automatically assign data download permissions to each node for the alliance chain role type;

[0008] Submit the data source to be labeled and the data labeling requirements corresponding to the data source to be labeled through the data labeling demander node, generate a work order number for the data source to be labeled, and generate a smart contract for the work order based on the data labeling requirements of the work order; convert a certain number of data sources to be labeled into a first data file in a random format, convert the data source to be labeled into a second data file in a random format, and generate a first key pair and a second key pair for the first data file and the second data file respectively; encrypt the first data file based on the first encryption key of the first key pair to form an encrypted first data file, and associate the encrypted first data file with the first decryption key; encrypt the second data file based on the second encryption key of the second key pair to form an encrypted second data file, and associate the encrypted second data file with the second decryption key; generate a third key pair for the work order, encrypt the first data file, the second data file, the smart contract and the data labeling requirements based on the third encryption key of the third key pair, and upload them together with the third decryption key to the data labeling work order block;

[0009] Download the data annotation work order block through the platform operation node and decrypt it, obtain the first data file, the first decryption key and the data annotation requirement, generate the work order dispatch data based on the first data file, the first decryption key and the data annotation requirement, generate the fourth key pair of the work order dispatch data through the encryption and decryption device, encrypt the work order dispatch data based on the fourth encryption key of the fourth key pair, and upload it to the work order dispatch block together with the fourth decryption key;

[0010] The work order dispatch block is downloaded through the data labeling operation party node, the second data file is decrypted, a fifth key pair is generated through the encryption and decryption device, and the decrypted data sources are encrypted one by one based on the encryption key of the fifth key pair. When the data labeling personnel perform the data labeling operation, the data sources are decrypted one by one for the data labeling personnel to operate to obtain the data labeling result, and each data source and the data labeling result corresponding to the data source are encrypted based on the fifth encryption key of the fifth key pair;

[0011] Determine whether the current work order has reached the work order submission time node. If so, submit the work order, generate a sixth key pair of data annotation results through the encryption and decryption device, encrypt the data annotation result file based on the sixth encryption key of the sixth key pair, and upload it together with the sixth decryption key to the work order data annotation result block;

[0012] Automatically download the work order data annotation result block and the data annotation work order block through the data annotation demander node, determine whether the current work order is the work order submitted by the current node, and if so, decrypt the data annotation result file based on the sixth decryption key, decrypt the data annotation results one by one based on the fifth decryption key, and mark the data annotation results one by one through the data demander, wherein the marking process includes confirmation, modification and deletion; then generate the seventh key pair through the encryption and decryption device, encrypt the data annotation results and the data annotation results after the marking process one by one, and encrypt the data annotation results based on the eighth encryption key of the eighth key pair;

[0013] After the data demander completes the acceptance, the smart contract will intelligently generate an acceptance report based on the quality assessment of the work order data annotation results;

[0014] Based on the acceptance report, the income of each data labeler corresponding to the work order is settled. After the income settlement is completed, the data demander is given the right to view the final data labeling results.

[0015] On the basis of the above technical solution, the present invention can also be improved as follows:

[0016] Furthermore, the data annotation requirements include the required completion time of data annotation, the field information to be annotated and extracted, and the relevant requirements for the extraction of corresponding field information annotation, accuracy requirements, acceptance criteria and payment rules:

[0017] The smart contracts include a work order completion time smart contract, a work order extraction field smart contract, a work order data annotation result quality assessment smart contract, and a work order fee freezing and payment smart contract;

[0018] A work order completion time smart contract is generated based on the data annotation completion time requirements; a work order extraction field smart contract is generated based on the field information extracted by the annotation and the corresponding field information annotation extraction related requirements; a work order data annotation result quality assessment smart contract is generated based on the accuracy requirements and acceptance criteria; and a work order fee freezing and payment smart contract is generated based on the payment rules.

[0019] Furthermore, generating work order dispatching data based on the first data file, the first decryption key and the data annotation requirement includes:

[0020] Calculating the first data file to obtain the complexity of the first data file;

[0021] Inputting the complexity of the first data file into the optimal work order annotation time prediction model to obtain the expected time range for completing the work order of the second data file;

[0022] The model is determined by the labeling personnel, the work order type of the second data file is determined, the historical work orders corresponding to the work order type are obtained, the data labeling personnel corresponding to the historical work orders are obtained, and the personnel to be assigned to the work orders are selected based on the data labeling personnel.

[0023] Furthermore, the generating of work order dispatching data based on the first data file, the first decryption key and the data annotation requirement further includes:

[0024] Get the total number of rows N of the data source to be annotated, and determine the row number sequence number of the data source to be annotated, randomly assign the data source to be annotated to M data annotators based on the row number sequence number, and generate a task data row sequence H for each data annotation data assigned to the work order task, and record it in the work order assignment data.

[0025] Further, the work order dispatching block is downloaded through the data labeling operation party node, the second data file is decrypted, a fifth key pair is generated through the encryption and decryption device, and the decrypted data sources are encrypted one by one based on the encryption key of the fifth key pair. When the data labeling personnel perform the data labeling operation, the data sources are decrypted one by one for the data labeling personnel to operate to obtain the data labeling result, and each data source and the data labeling result corresponding to the data source are encrypted based on the fifth encryption key of the fifth key pair, including:

[0026] During the operation, the data labeler decrypts the work order assigned data based on the fourth decryption key of the fourth key pair, decrypts the second data file, generates a fifth key pair based on the encryption and decryption device, encrypts the data source of the decrypted second data file one by one based on the encryption key of the fifth key pair, and then decrypts the data row sequence H assigned to the current data labeler row by row. When the data labeler completes the labeling of this row of data sources, each data labeling result is encrypted based on the fifth encryption key of the fifth key pair, encrypts this row of data sources again based on the fifth encryption key of the fifth key pair, and automatically decrypts the next row of data source display based on the fourth decryption key of the fifth key pair.

[0027] Furthermore, the acceptance report includes the quality assessment level of the work order data annotation results and the score of the data annotation personnel:

[0028] The quality assessment levels of the work order data annotation results are divided into level 0, level 1, level 2, level 3, level 4 and level 5;

[0029] When all the decrypted data labeling results are unqualified, the quality assessment level of the work order data labeling results is judged as level 0 or the data labeling personnel score corresponding to the data labeling results is judged as 0 points, and the data is destroyed on the data labeling demand side based on the work order data labeling result quality assessment smart contract.

[0030] Furthermore, after the data demander completes the acceptance, the smart contract for quality assessment based on the work order data annotation results intelligently generates an acceptance report, including:

[0031] The final revenue obtained by the current data annotation user is calculated by formula (1);

[0032] The final income of the current data annotation user = (H / N)*M*work order fee formula (1);

[0033] Where H is the number of data source rows assigned to the data annotation user corresponding to the current work order data annotation result, N is the total number of work order data source rows, and M is the revenue ratio of the work order data annotation result, which corresponds to the quality assessment level of the work order data annotation result.

[0034] A data annotation management system based on blockchain technology, comprising:

[0035] A construction module is used to construct a consortium chain system, wherein the consortium chain roles on the consortium chain system include platform operation nodes, data annotation demanders, and data annotation operators;

[0036] The permission allocation module is used to automatically allocate data download permissions for each node for the alliance chain role type based on the data security rules built by the alliance chain system;

[0037] A data annotation work order submission module is used to submit the data source to be annotated and the data annotation requirements corresponding to the data source to be annotated through the data annotation demander node, generate a work order number for the data source to be annotated, and generate a smart contract for the work order based on the data annotation requirements of the work order; convert a certain number of data sources to be annotated into a first data file in a random format, convert the data source to be annotated into a second data file in a random format, and generate a first key pair and a second key pair for the first data file and the second data file respectively; encrypt the first data file based on the first encryption key of the first key pair to form an encrypted first data file, and associate the encrypted first data file with the first decryption key; encrypt the second data file based on the second encryption key of the second key pair to form an encrypted second data file, and associate the encrypted second data file with the second decryption key; generate a third key pair for the work order, encrypt the first data file, the second data file, the smart contract and the data annotation requirements based on the third encryption key of the third key pair, and upload them together with the third decryption key to the data annotation work order block;

[0038] A work order assignment module is used to download and decrypt the data annotation work order block through the platform operation node, obtain a first data file, a first decryption key and a data annotation requirement, generate work order assignment data based on the first data file, the first decryption key and the data annotation requirement, generate a fourth key pair for the work order assignment data through an encryption and decryption device, encrypt the work order assignment data based on a fourth encryption key of the fourth key pair, and upload the work order assignment data together with the fourth decryption key to the work order assignment block;

[0039] A data labeling operation module, used to download the work order dispatching block through the data labeling operation party node, decrypt the second data file, generate a fifth key pair through the encryption and decryption device, and encrypt the decrypted data sources one by one based on the encryption key of the fifth key pair. When the data labeler performs the data labeling operation, the data sources are decrypted one by one for the data labeler to operate to obtain the data labeling result, and each data source and the data labeling result corresponding to the data source are encrypted based on the fifth encryption key of the fifth key pair;

[0040] The data labeling result submission module is used to determine whether the current work order has reached the work order submission time node. If so, submit the work order, generate a sixth key pair of the data labeling result through the encryption and decryption device, encrypt the data labeling result file based on the sixth encryption key of the sixth key pair, and upload it together with the sixth decryption key to the work order data labeling result block;

[0041] The data labeling result acceptance module is used to automatically download the work order data labeling result block and the data labeling work order block through the data labeling demand side node, determine whether the current work order is the work order submitted by the current node, and if so, decrypt the data labeling result file based on the sixth decryption key, decrypt the data labeling results one by one based on the fifth decryption key, and mark the data labeling results one by one through the data demand side, wherein the marking process includes confirmation, modification and deletion; then generate the seventh key pair through the encryption and decryption device, encrypt the data labeling results and the data labeling results after the marking process one by one, and encrypt the data labeling results based on the eighth encryption key of the eighth key pair;

[0042] The data annotation quality scoring module is used to intelligently generate an acceptance report based on the work order data annotation result quality assessment smart contract after the data demander completes the acceptance;

[0043] The data labeling work order payment module is used to settle the income of each data labeler corresponding to the work order based on the acceptance report. After the income settlement is completed, the data demander is given the right to view the final data labeling results.

[0044] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the steps of the method are implemented when the processor executes the computer program.

[0045] A non-transitory computer-readable storage medium stores a computer program, which implements the steps of the method when executed by a processor.

[0046] The embodiments of the present invention have the following advantages:

[0047] The data labeling management method based on blockchain technology in the present invention is constructed based on the underlying blockchain technology, and each operator of the data labeling business ecosystem forms an alliance chain system. Each participant in the data labeling business is a user node of the alliance chain. The data labeling management platform operates the user node and allocates block data download permissions based on data security rules and alliance chain node types. For example, for the data middle-end server node, due to platform operation needs, it is allocated data permissions such as data labeling work order block data, work order quality score block data, work order complexity analysis block data, work order completion time block data, and work order dispatch block data. The data labeling management system server node will automatically download the corresponding block data and perform decryption processing based on the corresponding decryption key, which solves the problem that the existing technology cannot ensure the data security of the labeling process and the data labeling business is not fair and rigorous enough. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0049] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0050] Figure 1 This is a flow chart of the data annotation management method based on blockchain technology of the present invention;

[0051] Figure 2 This is an architecture diagram of the data annotation management system based on blockchain technology of the present invention;

[0052] Figure 3 This is a schematic diagram of the physical structure of the electronic device provided by the present invention.

[0053] The accompanying drawings are denoted as follows:

[0054] Construction module 10, authority allocation module 20, data labeling work order submission module 30, work order allocation module 40, data labeling operation module 50, data labeling result submission module 60, data labeling result acceptance module 70, data labeling quality scoring module 80, data labeling work order payment module 90, electronic device 100, processor 1001, memory 1002, bus 1003. DETAILED DESCRIPTION

[0055] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] Figure 1 This is a flow chart of an embodiment of the data annotation management method based on blockchain technology of the present invention. Figure 1 As shown, a data annotation management method based on blockchain technology provided by an embodiment of the present invention includes the following steps:

[0057] S101, building a consortium chain system;

[0058] Specifically, the alliance chain roles in the alliance chain system include platform operation nodes, data labeling demanders, and data labeling operators.

[0059] S102, automatically assigning data download permissions to each node for the alliance chain role type based on the data security rules constructed by the alliance chain system;

[0060] S103, submitting the data source to be labeled and the data labeling requirements corresponding to the data source to be labeled through the data labeling demander node, generating a work order number for the data source to be labeled, and generating a smart contract for the work order based on the data labeling requirements of the work order; converting a certain number of data sources to be labeled into a first data file in a random format, converting the data source to be labeled into a second data file in a random format, and generating a first key pair and a second key pair for the first data file and the second data file respectively; encrypting the first data file based on the first encryption key of the first key pair to form an encrypted first data file, and associating the encrypted first data file with the first decryption key; encrypting the second data file based on the second encryption key of the second key pair to form an encrypted second data file, and associating the encrypted second data file with the second decryption key; generating a third key pair for the work order, encrypting the first data file, the second data file, the smart contract and the data labeling requirements based on the third encryption key of the third key pair, and uploading them together with the third decryption key to the data labeling work order block;

[0061] Specifically, the data annotation requirements include the required completion time of data annotation, the field information to be annotated and extracted, and the relevant requirements for the extraction of corresponding field information annotation, accuracy requirements, acceptance criteria and payment rules:

[0062] The smart contracts include a work order completion time smart contract, a work order extraction field smart contract, a work order data annotation result quality assessment smart contract, and a work order fee freezing and payment smart contract;

[0063] A work order completion time smart contract is generated based on the data annotation completion time requirements; a work order extraction field smart contract is generated based on the field information extracted by the annotation and the corresponding field information annotation extraction related requirements; a work order data annotation result quality assessment smart contract is generated based on the accuracy requirements and acceptance criteria; and a work order fee freezing and payment smart contract is generated based on the payment rules.

[0064] S104, downloading the data annotation work order block through the platform operation node and decrypting it, obtaining the first data file, the first decryption key and the data annotation requirement, generating the work order dispatch data based on the first data file, the first decryption key and the data annotation requirement, generating a fourth key pair for the work order dispatch data through the encryption and decryption device, encrypting the work order dispatch data based on the fourth encryption key of the fourth key pair, and uploading it together with the fourth decryption key to the work order dispatch block;

[0065] Specifically, calculating the first data file to obtain the complexity of the first data file;

[0066] The text content complexity is calculated by formula (4);

[0067] Text content complexity = (number of sentences / number of characters)*100 Formula (4);

[0068] The amount of text content is calculated by formula (5);

[0069] Text content = number of text characters / 1000;

[0070] The complexity of the first data file is obtained through entity relationship complexity, text content complexity and text content volume.

[0071] The complexity of the first data file is input into the optimal work order annotation time prediction model to obtain the expected time range for completing the work order of the second data file; the model is determined by the annotation personnel, the work order type of the second data file is determined, the historical work orders corresponding to the work order type are obtained, the data annotation personnel corresponding to the historical work orders are obtained, and the personnel to be assigned the work order are selected based on the data annotation personnel.

[0072] The labeling personnel determine the model, determine the work order type of the work order to be assigned, obtain the historical work order corresponding to the work order type, obtain the data labeling personnel corresponding to the historical work order, and determine the data labeling personnel with a quality level of 3 or above as the personnel to be assigned to the work order.

[0073] Analyze the annotators through test questions and obtain their scores based on different dimensions;

[0074] Obtain historical data of all labelers on the platform, and predict the types of labeling work orders that the labelers are good at based on the historical data;

[0075] The best work order type for the first assignment of the labeler is obtained based on the labeler's scores based on different dimensions and the labeling work order types that the labeler is good at.

[0076] The completion time node of the work order to be assigned to the personnel is calculated by formula (1);

[0077] Z1=Z0+Y1+Y2+X1 formula (1);

[0078] In the formula, Z1 is the completion time node of the person to be assigned the work order, Z0 is the current time node, Y1 is the intermittent rest time between work orders within a certain time period for the person to be assigned the work order, Y2 is the sum of the upper limit time of the time range required for the work order corresponding to the current work order to be processed by the person to be assigned the work order, and X1 is the upper limit of the expected time range for the work order to be assigned to complete the work order;

[0079] Determine whether Z1 exceeds the delivery time node Z required by the work order. If so, re-acquire the personnel to be assigned to the work order until the completion time node of the personnel to be assigned to the work order does not exceed the delivery time node required by the work order;

[0080] If not, the personnel to be assigned to the work order is stored in a pool of personnel to be screened and assigned, and it is determined whether the number of work order rows to be assigned is greater than a preset threshold value N1. If so, the personnel to be assigned to the work order with the fastest marking speed is selected from the pool of personnel to be screened and assigned;

[0081] Determine whether the number of work order data rows is less than a preset threshold value N2. If so, select the work order to-be-assigned personnel with the highest annotation quality level from the pool of to-be-screened assigned personnel, including:

[0082] When the number of work order rows is greater than the preset threshold value N1, the work order data of the work order type corresponding to the personnel to be assigned for each work order in the pool of personnel to be screened and assigned within a certain time period is obtained;

[0083] The average rate V of each work order type being processed by the personnel to be assigned to each work order is calculated by formula (6);

[0084] The average rate V of each work order type being processed by the personnel to be assigned to the work order = weighted average ((number of work order data rows * complexity of entity relationships of the work order * complexity of work order text content * amount of work order content) / actual time required to mark the work order) formula (6);

[0085] Select the person to be assigned the work order with the largest average rate V. If there are multiple people, select the person to be assigned the work order with the earliest completion time node;

[0086] When the number of work order rows is less than the preset threshold value N2, the work order data of the work order type corresponding to each work order to be assigned personnel in the pool of personnel to be screened and assigned within a certain period is obtained;

[0087] The average quality R of each work order type handled by the personnel to be assigned to each work order is calculated by formula (7);

[0088] Average quality R = weighted average (work order quality level) formula (7);

[0089] Select the person to be assigned the work order with the largest average quality R. If there are multiple people, select the person to be assigned the work order with the earliest completion time node.

[0090] The quality level of a certain number of historical work orders is obtained, including:

[0091] The modified quality value of the work order is calculated by formula (2);

[0092]

[0093] The work order deletion quality value is calculated by formula (3);

[0094]

[0095] For different work order types, when (total number of work order lines - number of canceled lines) is greater than the line number threshold and the work order deletion quality value is greater than or equal to the first threshold and the work order modification quality value is greater than or equal to the second threshold, the work order quality level is level 1;

[0096] For different work order types, when (total number of work order lines - number of canceled lines) is greater than the line number threshold and the work order deletion quality value is greater than or equal to the first threshold or the work order modification quality value is greater than or equal to the second threshold, the work order quality level is level 2;

[0097] For different work order types, when (total number of work order lines - number of canceled lines) is greater than the line number threshold and the work order deletion quality value is less than the first threshold and the work order modification quality value is less than the second threshold, the work order quality level is level 3;

[0098] For different work order types, when the work order deletion quality value is equal to 0 and the work order modification quality value is less than the second threshold or the work order modification quality value is equal to 0 and the work order deletion quality value is less than the first threshold, the work order quality level is level 4;

[0099] For different work order types, when the work order deletion quality value is equal to 0 and the work order modification quality value is equal to 0, the work order quality level is level 5.

[0100] Get the total number of rows N of the data source to be annotated, and determine the row number sequence number of the data source to be annotated, randomly assign the data source to be annotated to M data annotators based on the row number sequence number, and generate a task data row sequence H for each data annotation data assigned to the work order task, and record it in the work order assignment data.

[0101] S105, downloading the work order dispatching block through the data labeling operation party node, decrypting the second data file, generating a fifth key pair through the encryption and decryption device, encrypting the decrypted data sources one by one based on the encryption key of the fifth key pair, and when the data labeling personnel perform the data labeling operation, decrypting the data sources one by one for the data labeling personnel to operate, so as to obtain the data labeling result, and encrypting each data source and the data labeling result corresponding to the data source based on the fifth encryption key of the fifth key pair;

[0102] Specifically, during the operation, the data labeler decrypts the work order assigned data based on the fourth decryption key of the fourth key pair, decrypts the second data file, generates a fifth key pair based on the encryption and decryption device, encrypts the data source of the decrypted second data file one by one based on the encryption key of the fifth key pair, and then decrypts the data row sequence H (H1, H2, H3...HN) assigned to the current data labeler row by row. When the data labeler completes the labeling of this row of data sources, each data labeling result is encrypts based on the fifth encryption key of the fifth key pair, encrypts this row of data sources again based on the fifth encryption key of the fifth key pair, and automatically decrypts the next row of data source display based on the fifth decryption key of the fifth key pair.

[0103] So that data annotators can annotate the next row of data. In this way, the data source and the data annotation result data are secured. The data source is decrypted line by line, watermarked, annotated, the annotated data is encrypted line by line, and the annotated data source is encrypted line by line to ensure the security of the data source and the annotated data.

[0104] Based on the work order field extraction smart contract, a front-end display page is generated at the front end of the data labeler user node for the field information extracted from the work order annotation and the corresponding field information annotation extraction related requirements, which displays the corresponding field names and the related requirements for extracting field information, helping data labelers to complete data annotation more accurately and improve accuracy.

[0105] For example: the current work order extraction field intelligently extracts the business type field based on the annotation needs, and the relevant requirements for the annotation extraction of the corresponding field information contain noun definitions that describe the optional business type to ensure the uniformity of the business type field. The intelligent rule mechanism will generate intelligent rules based on this extraction field and extraction requirement: the components of the front-end business are displayed as drop-down boxes, and the drop-down options are the business type options in the annotation requirements.

[0106] S106, determining whether the current work order has reached the work order submission time node, if so, submitting the work order, generating a sixth key pair of data annotation results through the encryption and decryption device, encrypting the data annotation result file based on the sixth encryption key of the sixth key pair, and uploading it together with the sixth decryption key to the work order data annotation result block;

[0107] S107, automatically download the work order data annotation result block and the data annotation work order block through the data annotation demander node, determine whether the current work order is the work order submitted by the current node, and if so, decrypt the data annotation result file based on the sixth decryption key, decrypt the data annotation results one by one based on the fifth decryption key, and mark the data annotation results one by one through the data demander, wherein the marking process includes confirmation, modification and deletion; then generate the seventh key pair through the encryption and decryption device, encrypt the data annotation results and the data annotation results after the marking process one by one, and encrypt the data annotation results based on the eighth encryption key of the eighth key pair;

[0108] Specifically, encrypting the data labeling result based on the seventh encryption key of the seventh key pair can ensure the security of the labeled data before the work order is paid, and protect the rights and interests of the data labeling personnel.

[0109] S108, after the data demander completes the acceptance, the smart contract for quality assessment based on the work order data annotation results intelligently generates an acceptance report;

[0110] Specifically, the acceptance report includes the quality assessment level of the work order data annotation results and the score of the data annotation personnel:

[0111] The quality assessment levels of the work order data annotation results are divided into level 0, level 1, level 2, level 3, level 4 and level 5;

[0112] When all the decrypted data annotation results are unqualified, the quality assessment level of the work order data annotation results is judged as level 0 or the data annotation personnel score corresponding to the data annotation results is judged as 0 points, and the data annotation demander destroys the data based on the work order data annotation result quality assessment smart contract. (That is, the data annotation demander will not be able to see the data annotation result data submitted by the current data annotation personnel based on the work order). Resolve disputes arising in the data annotation business process. And the data annotation demander can repackage the data marked for deletion and initiate a data annotation work order.

[0113] When the acceptance score of the data annotation result submitted by the data annotation operator corresponding to the work order is 0, or the fee payment is completed after acceptance, the second data file data source and the data annotation result corresponding to the work order are destroyed at the data annotation operation node.

[0114] S109: Based on the acceptance report, the revenue of each data labeler corresponding to the work order is settled. After the revenue settlement is completed, the data demander is given the right to view the final data labeling results;

[0115] Specifically, the final revenue obtained by the current data annotation user is calculated by formula (1);

[0116] The final income of the current data annotation user = (H / N)*M*work order fee formula (1);

[0117] Where H is the number of data source rows assigned to the data annotation user corresponding to the current work order data annotation result, N is the total number of work order data source rows, and M is the revenue ratio of the work order data annotation result, which corresponds to the quality assessment level of the work order data annotation result.

[0118] Assume that the total number of rows of published work order data sources is N, the number of data source rows assigned to the data annotation user corresponding to the current work order data annotation result is H, and the work order data annotation result quality assessment level corresponding to the current data annotation user (the levels are divided into 6 levels: 0, 1, 2, 3, 4, 5, level 0 is 0 profit, level 1 is 20% profit, level 2 is 40% profit, level 3 is 60% profit, level 4 is 80% profit, level 5 is 100% profit, and the profit settlement methods are not currently limited to these). If the current work order data annotation result quality assessment level is level 3, the current data annotation user will eventually obtain profit = (H / N)*60%*work order fee.

[0119] Based on the work order fee freezing and payment smart contract, after the data labeling operator corresponding to the data labeling work order of the data demander completes the fee settlement, the final data result of the data labeling acceptance made by this data labeling operator is completely open to the data demander, that is, the seventh key pair automatically completes the decryption process, and the data demander can view the corresponding data at any time, thereby safeguarding the rights and interests of all parties in the data labeling transaction process.

[0120] The data annotation management method based on blockchain technology builds an alliance chain system, wherein the alliance chain roles on the alliance chain system include platform operation nodes, data annotation demanders and data annotation operators; data download permissions of each node are automatically allocated to the alliance chain role type based on the data security rules constructed by the alliance chain system; a data source to be annotated and a data annotation requirement corresponding to the data source to be annotated are submitted by the data annotation demander node, a work order number is generated for the data source to be annotated, and a smart contract for the work order is generated based on the data annotation requirement of the work order; a certain number of data sources to be annotated are converted into a first data file in a random format, and the data source to be annotated is converted into a A second data file in a random format, generating a first key pair and a second key pair for the first data file and the second data file respectively; encrypting the first data file based on the first encryption key of the first key pair to form an encrypted first data file, and associating the encrypted first data file with the first decryption key; encrypting the second data file based on the second encryption key of the second key pair to form an encrypted second data file, and associating the encrypted second data file with the second decryption key; generating a third key pair for the work order, and encrypting the first data file, the second data file, the smart contract, and the data annotation requirements based on the third encryption key of the third key pair. The first data file, the first decryption key and the data annotation requirement are obtained, and the work order dispatch data is generated based on the first data file, the first decryption key and the data annotation requirement, and a fourth key pair of the work order dispatch data is generated through the encryption and decryption device, and the work order dispatch data is encrypted based on the fourth encryption key of the fourth key pair, and uploaded to the work order dispatch block together with the fourth decryption key; the work order dispatch block is downloaded through the data annotation operation party node, the second data file is decrypted, and a fifth key pair is generated through the encryption and decryption device, The decrypted data sources are encrypted one by one based on the encryption key of the fifth key pair. When the data labeler performs the data labeling operation, the data sources are decrypted one by one for the data labeler to perform the operation, so as to obtain the data labeling result. Each data source and the data labeling result corresponding to the data source are encrypted based on the fifth encryption key of the fifth key pair; it is determined whether the current work order has reached the work order submission time node. If so, the work order is submitted, and the sixth key pair of the data labeling result is generated by the encryption and decryption device. The data labeling result file is encrypted based on the sixth encryption key of the sixth key pair, and uploaded to the work order data labeling result block together with the sixth decryption key;The data annotation demander node automatically downloads the work order data annotation result block and the data annotation work order block, determines whether the current work order is the work order submitted by the current node, and if so, decrypts the data annotation result file based on the sixth decryption key, decrypts the data annotation results one by one based on the fifth decryption key, and marks the data annotation results one by one through the data demander, wherein the marking process includes confirmation, modification and deletion; then generates the seventh key pair through the encryption and decryption device, encrypts the data annotation results and the data annotation results after the marking process one by one, and encrypts the data annotation results based on the eighth encryption key of the eighth key pair; after the data demander completes the acceptance, the acceptance report is intelligently generated based on the work order data annotation result quality assessment smart contract; based on the acceptance report, the income of each data annotation personnel corresponding to the work order is settled, and after the income settlement is completed, the data demander is given the right to view the final data annotation results. It solves the problem that the existing technology cannot guarantee the data security of the annotation process and the data annotation business is not fair and rigorous. ;

[0121] Figure 2 This is an architecture diagram of an embodiment of a data annotation management system based on blockchain technology of the present invention; Figure 2 As shown, an embodiment of the present invention provides a data annotation management system based on blockchain technology, comprising the following steps:

[0122] A construction module 10 is used to construct a consortium chain system, wherein the consortium chain roles on the consortium chain system include platform operation nodes, data annotation demanders, and data annotation operators;

[0123] The permission allocation module 20 is used to automatically allocate data download permissions for each node for the alliance chain role type based on the data security rules constructed by the alliance chain system;

[0124] The data annotation work order submission module 30 is used to submit the data source to be annotated and the data annotation requirements corresponding to the data source to be annotated through the data annotation demander node, generate a work order number for the data source to be annotated, and generate a smart contract for the work order based on the data annotation requirements of the work order; convert a certain number of data sources to be annotated into a first data file in a random format, convert the data source to be annotated into a second data file in a random format, and generate a first key pair and a second key pair for the first data file and the second data file respectively; encrypt the first data file based on the first encryption key of the first key pair to form an encrypted first data file, and associate the encrypted first data file with the first decryption key; encrypt the second data file based on the second encryption key of the second key pair to form an encrypted second data file, and associate the encrypted second data file with the second decryption key; generate a third key pair for the work order, encrypt the first data file, the second data file, the smart contract and the data annotation requirements based on the third encryption key of the third key pair, and upload them together with the third decryption key to the data annotation work order block;

[0125] The data annotation requirements include the required completion time of data annotation, the field information to be annotated and extracted, and the relevant requirements for the extraction of corresponding field information annotation, accuracy requirements, acceptance criteria and payment rules:

[0126] The smart contracts include a work order completion time smart contract, a work order extraction field smart contract, a work order data annotation result quality assessment smart contract, and a work order fee freezing and payment smart contract;

[0127] A work order completion time smart contract is generated based on the data annotation completion time requirements; a work order extraction field smart contract is generated based on the field information extracted by the annotation and the corresponding field information annotation extraction related requirements; a work order data annotation result quality assessment smart contract is generated based on the accuracy requirements and acceptance criteria; and a work order fee freezing and payment smart contract is generated based on the payment rules.

[0128] The work order assignment module 40 is used to download and decrypt the data annotation work order block through the platform operation node, obtain the first data file, the first decryption key and the data annotation requirement, generate the work order assignment data based on the first data file, the first decryption key and the data annotation requirement, generate a fourth key pair for the work order assignment data through the encryption and decryption device, encrypt the work order assignment data based on the fourth encryption key of the fourth key pair, and upload it to the work order assignment block together with the fourth decryption key;

[0129] The work order allocation module 40 is also used for:

[0130] Calculating the first data file to obtain the complexity of the first data file;

[0131] Inputting the complexity of the first data file into the optimal work order annotation time prediction model to obtain the expected time range for completing the work order of the second data file;

[0132] The model is determined by the labeling personnel, the work order type of the second data file is determined, the historical work orders corresponding to the work order type are obtained, the data labeling personnel corresponding to the historical work orders are obtained, and the personnel to be assigned to the work orders are selected based on the data labeling personnel.

[0133] Get the total number of rows N of the data source to be annotated, and determine the row number sequence number of the data source to be annotated, randomly assign the data source to be annotated to M data annotators based on the row number sequence number, and generate a task data row sequence H for each data annotation data assigned to the work order task, and record it in the work order assignment data.

[0134] The data labeling operation module 50 is used to download the work order dispatching block through the data labeling operation party node, decrypt the second data file, generate a fifth key pair through the encryption and decryption device, and encrypt the decrypted data sources one by one based on the encryption key of the fifth key pair. When the data labeler performs the data labeling operation, the data sources are decrypted one by one for the data labeler to operate to obtain the data labeling result, and each data source and the data labeling result corresponding to the data source are encrypted based on the fifth encryption key of the fifth key pair;

[0135] The data annotation module 50 is also used for:

[0136] During the operation, the data labeler decrypts the work order assigned data based on the fourth decryption key of the fourth key pair, decrypts the second data file, generates a fifth key pair based on the encryption and decryption device, encrypts the data source of the decrypted second data file one by one based on the encryption key of the fifth key pair, and then decrypts the data row sequence H assigned to the current data labeler row by row. When the data labeler completes the labeling of this row of data sources, each data labeling result is encrypted based on the fifth encryption key of the fifth key pair, encrypts this row of data sources again based on the fifth encryption key of the fifth key pair, and automatically decrypts the next row of data source display based on the fourth decryption key of the fifth key pair.

[0137] The data labeling result submission module 60 is used to determine whether the current work order has reached the work order submission time node. If so, submit the work order, generate a sixth key pair of the data labeling result through the encryption and decryption device, encrypt the data labeling result file based on the sixth encryption key of the sixth key pair, and upload it together with the sixth decryption key to the work order data labeling result block;

[0138] The data labeling result acceptance module 70 is used to automatically download the work order data labeling result block and the data labeling work order block through the data labeling demand side node, determine whether the current work order is the work order submitted by the current node, and if so, decrypt the data labeling result file based on the sixth decryption key, decrypt the data labeling results one by one based on the fifth decryption key, and mark the data labeling results one by one through the data demand side, wherein the marking process includes confirmation, modification and deletion; then generate the seventh key pair through the encryption and decryption device, encrypt the data labeling results and the data labeling results after the marking process one by one, and encrypt the data labeling results based on the eighth encryption key of the eighth key pair;

[0139] The data annotation quality scoring module 80 is used to intelligently generate an acceptance report based on the work order data annotation result quality assessment smart contract after the data demander completes the acceptance;

[0140] The acceptance report includes the quality assessment level of the work order data annotation results and the score of the data annotator:

[0141] The quality assessment levels of the work order data annotation results are divided into level 0, level 1, level 2, level 3, level 4 and level 5;

[0142] When all the decrypted data labeling results are unqualified, the quality assessment level of the work order data labeling results is judged as level 0 or the data labeling personnel score corresponding to the data labeling results is judged as 0 points, and the data is destroyed on the data labeling demand side based on the work order data labeling result quality assessment smart contract.

[0143] The data labeling work order payment module 90 is used to settle the income of each data labeling personnel corresponding to the work order based on the acceptance report. After the income settlement is completed, the data demander is given the right to view the final data labeling results.

[0144] The data annotation work order payment module 90 is also used for:

[0145] The final revenue obtained by the current data annotation user is calculated by formula (1);

[0146] The final income of the current data annotation user = (H / N)*M*work order fee formula (1);

[0147] Where H is the number of data source rows assigned to the data annotation user corresponding to the current work order data annotation result, N is the total number of work order data source rows, and M is the revenue ratio of the work order data annotation result, which corresponds to the quality assessment level of the work order data annotation result.

[0148] Figure 3 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention, such as Figure 3 As shown, the electronic device 100 includes: a processor 1001 (processor), a memory 1002 (memory) and a bus 1003; wherein the processor 1001 and the memory 1002 communicate with each other through the bus 1003;

[0149] The processor 1001 is used to call the program instructions in the memory 1002 to execute the methods provided by the above-mentioned method embodiments, for example, including: building a consortium chain system, wherein the consortium chain roles on the consortium chain system include platform operation nodes, data labeling demanders and data labeling operators; automatically allocating data download permissions for each node for the consortium chain role type based on the data security rules constructed by the consortium chain system; submitting the data source to be labeled and the data labeling requirements corresponding to the data source to be labeled through the data labeling demander node, generating a work order number for the data source to be labeled, and generating a smart contract for the work order based on the data labeling requirements of the work order; converting a certain number of data sources to be labeled into random The first data file in a random format is converted into a second data file in a random format, and a first key pair and a second key pair are generated for the first data file and the second data file respectively; the first data file is encrypted based on the first encryption key of the first key pair to form an encrypted first data file, and the encrypted first data file is associated with the first decryption key; the second data file is encrypted based on the second encryption key of the second key pair to form an encrypted second data file, and the encrypted second data file is associated with the second decryption key; a third key pair is generated for the work order, and the first data file and the second data file are encrypted based on the third encryption key of the third key pair. The first data file, the first decryption key and the data annotation requirement are encrypted, and uploaded to the data annotation work order block together with the third decryption key; the data annotation work order block is downloaded and decrypted through the platform operation node, and the first data file, the first decryption key and the data annotation requirement are obtained, and the work order dispatch data is generated based on the first data file, the first decryption key and the data annotation requirement, and the fourth key pair of the work order dispatch data is generated through the encryption and decryption device, and the work order dispatch data is encrypted based on the fourth encryption key of the fourth key pair, and uploaded to the work order dispatch block together with the fourth decryption key; the work order dispatch block is downloaded through the data annotation operation party node, the second data file is decrypted, and the encryption and decryption device is used to generate the work order dispatch data. A fifth key pair is formed, and the decrypted data sources are encrypted one by one based on the encryption key of the fifth key pair. When the data labeling personnel perform the data labeling operation, the data sources are decrypted one by one for the data labeling personnel to operate, so as to obtain the data labeling result. Each data source and the data labeling result corresponding to the data source are encrypted based on the fifth encryption key of the fifth key pair; it is determined whether the current work order has reached the work order submission time node. If so, the work order is submitted, and a sixth key pair of the data labeling result is generated by the encryption and decryption device. The data labeling result file is encrypted based on the sixth encryption key of the sixth key pair, and uploaded to the work order data labeling result block together with the sixth decryption key;The data annotation demander node automatically downloads the work order data annotation result block and the data annotation work order block, determines whether the current work order is the work order submitted by the current node, and if so, decrypts the data annotation result file based on the sixth decryption key, decrypts the data annotation results one by one based on the fifth decryption key, and performs marking processing on the data annotation results one by one through the data demander, wherein the marking processing includes confirmation, modification and deletion; then generates the seventh key pair through the encryption and decryption device, encrypts the data annotation results and the data annotation results after the marking processing one by one, and encrypts the data annotation results based on the eighth encryption key of the eighth key pair; after the data demander completes the acceptance, the acceptance report is intelligently generated based on the work order data annotation result quality assessment smart contract; the income of each data annotation personnel corresponding to the work order is settled based on the acceptance report, and after the income settlement is completed, the data demander is given the right to view the final data annotation results. ;

[0150] The present embodiment provides a non-transitory computer-readable storage medium, which stores computer instructions. The computer instructions enable a computer to execute the methods provided by the above-mentioned method embodiments, for example, including: building a consortium chain system, wherein the consortium chain roles on the consortium chain system include platform operation nodes, data labeling demanders, and data labeling operators; automatically allocating data download permissions to each node for the consortium chain role type based on the data security rules constructed by the consortium chain system; submitting the data source to be labeled and the data labeling requirements corresponding to the data source to be labeled through the data labeling demander node, generating a work order number for the data source to be labeled, and generating an intelligent consortium chain based on the data labeling requirements of the work order. The method comprises the following steps: converting a certain number of data sources to be annotated into a first data file in a random format, converting the data source to be annotated into a second data file in a random format, generating a first key pair and a second key pair for the first data file and the second data file, respectively; encrypting the first data file based on a first encryption key of the first key pair to form an encrypted first data file, and associating the encrypted first data file with a first decryption key; encrypting the second data file based on a second encryption key of the second key pair to form an encrypted second data file, and associating the encrypted second data file with a second decryption key; generating a third key pair for the work order, and generating a third encryption key pair based on a third encryption key pair of the third key pair. The first data file, the second data file, the smart contract and the data annotation requirement are encrypted by the key, and uploaded to the data annotation work order block together with the third decryption key; the data annotation work order block is downloaded and decrypted through the platform operation node, the first data file, the first decryption key and the data annotation requirement are obtained, and the work order dispatch data is generated based on the first data file, the first decryption key and the data annotation requirement, and the fourth key pair of the work order dispatch data is generated through the encryption and decryption device, and the work order dispatch data is encrypted based on the fourth encryption key of the fourth key pair, and uploaded to the work order dispatch block together with the fourth decryption key; the work order dispatch block is downloaded through the data annotation operation party node, and the second data file is decrypted. Generate a fifth key pair through the encryption and decryption device, encrypt the decrypted data sources one by one based on the encryption key of the fifth key pair, and when the data labeling personnel perform data labeling operations, decrypt the data sources one by one for the data labeling personnel to perform operations to obtain data labeling results, and encrypt each data source and the data labeling result corresponding to the data source based on the fifth encryption key of the fifth key pair; determine whether the current work order has reached the work order submission time node, if so, submit the work order, generate a sixth key pair of the data labeling result through the encryption and decryption device, encrypt the data labeling result based on the sixth encryption key of the sixth key pair, and upload it to the work order data labeling result block together with the sixth decryption key;The data annotation demander node automatically downloads the work order data annotation result block and the data annotation work order block, determines whether the current work order is the work order submitted by the current node, and if so, decrypts the data annotation result file based on the sixth decryption key, decrypts the data annotation results one by one based on the fifth decryption key, and performs marking processing on the data annotation results one by one through the data demander, wherein the marking processing includes confirmation, modification and deletion; then generates the seventh key pair through the encryption and decryption device, encrypts the data annotation results and the data annotation results after the marking processing one by one, and encrypts the data annotation results based on the eighth encryption key of the eighth key pair; after the data demander completes the acceptance, the acceptance report is intelligently generated based on the work order data annotation result quality assessment smart contract; the income of each data annotation personnel corresponding to the work order is settled based on the acceptance report, and after the income settlement is completed, the data demander is given the right to view the final data annotation results. ;

[0151] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various storage media that can store program codes.

[0152] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0153] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0154] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A data annotation management method based on blockchain technology, characterized in that: include: Constructing a consortium chain system, wherein the consortium chain roles on the consortium chain system include platform operation nodes, data annotation demanders, and data annotation operators; The data security rules built based on the alliance chain system automatically assign data download permissions to each node for the alliance chain role type; Submit the data source to be labeled and the data labeling requirements corresponding to the data source to be labeled through the data labeling demander node, generate a work order number for the data source to be labeled, and generate a smart contract for the work order based on the data labeling requirements of the work order; convert a certain number of data sources to be labeled into a first data file in a random format, convert the data source to be labeled into a second data file in a random format, and generate a first key pair and a second key pair for the first data file and the second data file respectively; encrypt the first data file based on the first encryption key of the first key pair to form an encrypted first data file, and associate the encrypted first data file with the first decryption key; encrypt the second data file based on the second encryption key of the second key pair to form an encrypted second data file, and associate the encrypted second data file with the second decryption key; generate a third key pair for the work order, encrypt the first data file, the second data file, the smart contract and the data labeling requirements based on the third encryption key of the third key pair, and upload them together with the third decryption key to the data labeling work order block; Download the data annotation work order block through the platform operation node and decrypt it, obtain the first data file, the first decryption key and the data annotation requirement, generate the work order dispatch data based on the first data file, the first decryption key and the data annotation requirement, generate the fourth key pair of the work order dispatch data through the encryption and decryption device, encrypt the work order dispatch data based on the fourth encryption key of the fourth key pair, and upload it to the work order dispatch block together with the fourth decryption key; The work order dispatch block is downloaded through the data labeling operation party node, the second data file is decrypted, a fifth key pair is generated through the encryption and decryption device, and the decrypted data sources are encrypted one by one based on the encryption key of the fifth key pair. When the data labeling personnel perform the data labeling operation, the data sources are decrypted one by one for the data labeling personnel to operate to obtain the data labeling result, and each data source and the data labeling result corresponding to the data source are encrypted based on the fifth encryption key of the fifth key pair; Determine whether the current work order has reached the work order submission time node. If so, submit the work order, generate a sixth key pair of data annotation results through the encryption and decryption device, encrypt the data annotation result file based on the sixth encryption key of the sixth key pair, and upload it together with the sixth decryption key to the work order data annotation result block; Automatically download the work order data annotation result block and the data annotation work order block through the data annotation demander node, determine whether the current work order is the work order submitted by the current node, and if so, decrypt the data annotation result file based on the sixth decryption key, decrypt the data annotation results one by one based on the fifth decryption key, and mark the data annotation results one by one through the data demander, wherein the marking process includes confirmation, modification and deletion; then generate the seventh key pair through the encryption and decryption device, encrypt the data annotation results and the data annotation results after the marking process one by one, and encrypt the data annotation results based on the eighth encryption key of the eighth key pair; After the data demander completes the acceptance, the smart contract will intelligently generate an acceptance report based on the quality assessment of the work order data annotation results; Based on the acceptance report, the income of each data labeler corresponding to the work order is settled. After the income settlement is completed, the data demander is given the right to view the final data labeling results.

2. According to the data annotation management method based on blockchain technology according to claim 1, it is characterized in that: The data annotation requirements include the required completion time of data annotation, the field information to be annotated and extracted, and the relevant requirements for the extraction of corresponding field information annotation, accuracy requirements, acceptance criteria and payment rules: The smart contracts include a work order completion time smart contract, a work order extraction field smart contract, a work order data annotation result quality assessment smart contract, and a work order fee freezing and payment smart contract; A work order completion time smart contract is generated based on the data annotation completion time requirements; a work order extraction field smart contract is generated based on the field information extracted by the annotation and the corresponding field information annotation extraction related requirements; a work order data annotation result quality assessment smart contract is generated based on the accuracy requirements and acceptance criteria; and a work order fee freezing and payment smart contract is generated based on the payment rules.

3. According to the data annotation management method based on blockchain technology according to claim 1, it is characterized in that: The generating work order dispatching data based on the first data file, the first decryption key and the data annotation requirement includes: Calculating the first data file to obtain the complexity of the first data file; Inputting the complexity of the first data file into the optimal work order annotation time prediction model to obtain the expected time range for completing the work order of the second data file; The model is determined by the labeling personnel, the work order type of the second data file is determined, the historical work orders corresponding to the work order type are obtained, the data labeling personnel corresponding to the historical work orders are obtained, and the personnel to be assigned to the work orders are selected based on the data labeling personnel.

4. According to the data annotation management method based on blockchain technology according to claim 3, it is characterized in that: The step of generating work order dispatch data based on the first data file, the first decryption key and the data annotation requirement further includes: Get the total number of rows N of the data source to be annotated, and determine the row number sequence number of the data source to be annotated, randomly assign the data source to be annotated to M data annotators based on the row number sequence number, and generate a task data row sequence H for each data annotation data assigned to the work order task, and record it in the work order assignment data.

5. According to the data annotation management method based on blockchain technology according to claim 1, it is characterized in that: The work order dispatching block is downloaded through the data labeling operation party node, the second data file is decrypted, a fifth key pair is generated through the encryption and decryption device, and the decrypted data sources are encrypted one by one based on the encryption key of the fifth key pair. When the data labeling personnel perform the data labeling operation, the data sources are decrypted one by one for the data labeling personnel to operate to obtain the data labeling result, and each data source and the data labeling result corresponding to the data source are encrypted based on the fifth encryption key of the fifth key pair, including: During the operation, the data labeler decrypts the work order assigned data based on the fourth decryption key of the fourth key pair, decrypts the second data file, generates a fifth key pair based on the encryption and decryption device, encrypts the data source of the decrypted second data file one by one based on the encryption key of the fifth key pair, and then decrypts the data row sequence H assigned to the current data labeler row by row. When the data labeler completes the labeling of this row of data sources, each data labeling result is encrypted based on the fifth encryption key of the fifth key pair, encrypts this row of data sources again based on the fifth encryption key of the fifth key pair, and automatically decrypts the next row of data source display based on the fourth decryption key of the fifth key pair.

6. According to the data annotation management method based on blockchain technology according to claim 1, it is characterized in that: The acceptance report includes the quality assessment level of the work order data annotation results and the score of the data annotator: The quality assessment levels of the work order data annotation results are divided into level 0, level 1, level 2, level 3, level 4 and level 5; When all the decrypted data labeling results are unqualified, the quality assessment level of the work order data labeling results is judged as level 0 or the data labeling personnel score corresponding to the data labeling results is judged as 0 points, and the data is destroyed on the data labeling demand side based on the work order data labeling result quality assessment smart contract.

7. According to the data annotation management method based on blockchain technology according to claim 1, it is characterized in that: After the data demander completes the acceptance, the smart contract for quality assessment based on the work order data annotation results intelligently generates an acceptance report, including: The final revenue obtained by the current data annotation user is calculated by formula (1); The final income of the current data annotation user = (H / N)*M*work order fee formula (1); Where H is the number of data source rows assigned to the data annotation user corresponding to the current work order data annotation result, N is the total number of work order data source rows, and M is the revenue ratio of the work order data annotation result, which corresponds to the quality assessment level of the work order data annotation result.

8. A data annotation management system based on blockchain technology, characterized in that: include: A construction module is used to construct a consortium chain system, wherein the consortium chain roles on the consortium chain system include platform operation nodes, data annotation demanders, and data annotation operators; The permission allocation module is used to automatically allocate data download permissions for each node for the alliance chain role type based on the data security rules built by the alliance chain system; A data annotation work order submission module is used to submit the data source to be annotated and the data annotation requirements corresponding to the data source to be annotated through the data annotation demander node, generate a work order number for the data source to be annotated, and generate a smart contract for the work order based on the data annotation requirements of the work order; convert a certain number of data sources to be annotated into a first data file in a random format, convert the data source to be annotated into a second data file in a random format, and generate a first key pair and a second key pair for the first data file and the second data file respectively; encrypt the first data file based on the first encryption key of the first key pair to form an encrypted first data file, and associate the encrypted first data file with the first decryption key; encrypt the second data file based on the second encryption key of the second key pair to form an encrypted second data file, and associate the encrypted second data file with the second decryption key; generate a third key pair for the work order, encrypt the first data file, the second data file, the smart contract and the data annotation requirements based on the third encryption key of the third key pair, and upload them together with the third decryption key to the data annotation work order block; A work order assignment module is used to download and decrypt the data annotation work order block through the platform operation node, obtain a first data file, a first decryption key and a data annotation requirement, generate work order assignment data based on the first data file, the first decryption key and the data annotation requirement, generate a fourth key pair for the work order assignment data through an encryption and decryption device, encrypt the work order assignment data based on a fourth encryption key of the fourth key pair, and upload the work order assignment data together with the fourth decryption key to the work order assignment block; A data labeling operation module, used to download the work order dispatching block through the data labeling operation party node, decrypt the second data file, generate a fifth key pair through the encryption and decryption device, and encrypt the decrypted data sources one by one based on the encryption key of the fifth key pair. When the data labeler performs the data labeling operation, the data sources are decrypted one by one for the data labeler to operate to obtain the data labeling result, and each data source and the data labeling result corresponding to the data source are encrypted based on the fifth encryption key of the fifth key pair; The data labeling result submission module is used to determine whether the current work order has reached the work order submission time node. If so, submit the work order, generate a sixth key pair of the data labeling result through the encryption and decryption device, encrypt the data labeling result file based on the sixth encryption key of the sixth key pair, and upload it together with the sixth decryption key to the work order data labeling result block; The data labeling result acceptance module is used to automatically download the work order data labeling result block and the data labeling work order block through the data labeling demander node, determine whether the current work order is the work order submitted by the current node, and if so, decrypt the data labeling result file based on the sixth decryption key, decrypt the data labeling results one by one based on the fifth decryption key, and mark the data labeling results one by one through the data demander, wherein the marking process includes confirmation, modification and deletion; then generate the seventh key pair through the encryption and decryption device, encrypt the data labeling results and the data labeling results after the marking process one by one, and encrypt the data labeling results based on the eighth encryption key of the eighth key pair; The data annotation quality scoring module is used to intelligently generate an acceptance report based on the work order data annotation result quality assessment smart contract after the data demander completes the acceptance; The data labeling work order payment module is used to settle the income of each data labeler corresponding to the work order based on the acceptance report. After the income settlement is completed, the data demander is given the right to view the final data labeling results.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A non-transitory computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images