Data annotation cost measurement method and system, electronic equipment and storage medium

Abstract

The invention relates to the technical field of computers, and discloses a data annotation cost measurement method and system, electronic equipment and a storage medium, and the method comprises the steps: determining a to-be-evaluated digital processing flow range; formulating an inspection-free standard for judging whether the data output by the automatic algorithm meets the quality requirement or not so as to exempt manual inspection; a check item weight is allocated to each check item in the digital processing flow range, the sum of the weights of all the check items is a fixed value, and an operation weight is allocated to the manual operation type; calculating a data exemption rate processed by the automatic algorithm based on an exemption standard; and calculating the auxiliary processing cost of the automatic algorithm based on the inspection item weight and the operation weight, calculating the total manual processing cost based on the inspection item weight and the modification operation weight, and calculating the cost saving index according to the proportional relation between the auxiliary processing cost and the total manual processing cost. The method can achieve the precision measurement of the economic value of the algorithm from the perspective of the cost structure.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and specifically to a data annotation cost measurement method, system, electronic device, and storage medium. Background Technology

[0002] With the application of artificial intelligence technology, automated annotation algorithms (such as optical character recognition and automatic indexing) have been widely introduced into data production lines, aiming to replace manual labor, improve efficiency, and reduce costs. However, the industry currently faces significant bottlenecks in accurately measuring the economic benefits of these algorithms.

[0003] Existing evaluation systems primarily focus on technical performance metrics such as accuracy and recall of algorithms. While these metrics reflect the technical level of an algorithm, they cannot be directly translated into cost savings and return on investment required for business decisions. As a result, project managers find it difficult to quantify how much labor costs the introduction of a new algorithm can reduce, leading to a lack of clear economic basis for technology investment decisions.

[0004] Existing solutions only count the number of errors in the algorithm results, ignoring the significant differences in complexity and time consumption of the manual operations required to correct different errors. For example, handling some errors may only require simple confirmation, while handling others may involve complex analysis, reconstruction, or decision-making. Due to the lack of refined modeling of "operation types" and their corresponding "cost weights," existing technologies cannot accurately measure the economic value of algorithms from a cost structure perspective, thus creating a seemingly insurmountable "evaluation gap" between algorithm performance improvement and its economic value assessment.

[0005] Therefore, this application provides a data annotation cost measurement method to solve the above-mentioned technical problems. Summary of the Invention

[0006] The purpose of this invention is to provide a data annotation cost measurement method, system, electronic device and storage medium to solve the technical problem that the existing technology cannot accurately measure the economic value of the algorithm from the perspective of cost structure.

[0007] To address the aforementioned technical problems, this invention provides a data annotation cost measurement method, comprising:

[0008] Define the scope of the digital processing workflow to be evaluated, including clarifying the data format, data size, annotation rules, and quality requirements based on the annotation task description document;

[0009] Establish inspection exemption standards to determine whether the data output by the automated algorithm meets the quality requirements and is exempt from manual inspection. The inspection exemption standards are established with reference to the benchmark qualification standards and the pre-labeling results of the automated algorithm.

[0010] Each inspection item within the scope of the digital processing flow is assigned an inspection item weight, and the sum of the weights of all inspection items is a fixed value. At the same time, an operation weight is assigned to the manual operation type, wherein the operation weight is positively correlated with the comprehensive cost of the corresponding operation.

[0011] The data exemption rate after processing by the automated algorithm is calculated based on the exemption standard, wherein the data exemption rate is determined based on the ratio of the number of exempted data entries to the total number of data entries;

[0012] The auxiliary processing cost of the automated algorithm is calculated based on the inspection item weight and the operation weight. The total manual processing cost is calculated based on the inspection item weight and the modification operation weight. The cost saving index is calculated based on the ratio between the auxiliary processing cost and the total manual processing cost.

[0013] In some specific embodiments, the scope of the digitization process to be evaluated is determined, wherein the data format, data scale, annotation rules, and quality requirements are clarified based on the annotation task specification document, and further includes:

[0014] Parse the annotation task description document to extract information on data storage format, file type, and estimated total data volume;

[0015] Based on the annotation specifications in the annotation task description document, the scope of the inspection items that need to be evaluated is defined, and a set of inspection items is formed;

[0016] Determine the annotation rules, definitions of professional terms, and descriptions of various quality requirements specified in the annotation task description document;

[0017] Establish a quality indicator system, including accuracy, completeness, and consistency indicators, as well as corresponding acceptable threshold ranges.

[0018] In some specific embodiments, an exemption standard is established to determine whether the data output by the automated algorithm meets the quality requirements and is exempt from manual inspection. The exemption standard is established with reference to the benchmark acceptance standard and the pre-labeling results of the automated algorithm, and further includes:

[0019] Establish baseline acceptance criteria for all data outputs;

[0020] Analyze the pre-labeling results of automated algorithms on sample datasets, where the pre-labeling results include accuracy and recall metrics;

[0021] Based on the analysis of the aforementioned benchmark qualification standards and pre-labeling results, the parameters for exemption from inspection and the judgment rules are determined;

[0022] The aforementioned exemption criteria parameters and judgment rules are integrated.

[0023] In some specific embodiments, inspection item weights are assigned to each inspection item within the scope of the digital processing flow, and the sum of the weights of all inspection items is a fixed value. Simultaneously, operation weights are assigned to manual operation types, wherein the operation weights are positively correlated with the overall cost of the corresponding operation, further including:

[0024] The weights of each inspection item are assigned according to its business importance and frequency of use in the data application scenario, or according to the historical data proportion of each inspection item in the piece-rate wage calculation system.

[0025] The weights of all inspection items are accumulated through normalization to reach the fixed value;

[0026] Maintain consistency between the weighting ratio of inspection items and the billing ratio in actual production cost accounting.

[0027] In some specific embodiments, operation weights are assigned to manual operation types, wherein the operation weights are positively correlated with the overall cost of the corresponding operation, and further include:

[0028] Define the set of operation types performed by human operators in an automated algorithm-assisted environment;

[0029] Analyze the factors influencing the time cost, skill requirements, and operational complexity for each type of operation;

[0030] The relative weight values ​​of each operation type relative to the baseline operation are determined based on the comprehensive analysis results;

[0031] A comparable operation weight system is established between different operation types through standardization.

[0032] In some specific embodiments, the data exemption rate after processing by the automated algorithm is calculated based on the exemption standard, wherein the data exemption rate is determined based on the ratio of the number of exempted data entries to the total number of data entries, and further includes:

[0033] Within the set statistical time period, obtain the number of data entries automatically determined to be exempt from inspection;

[0034] Within the same statistical time period, obtain the total number of data entries that have been processed by automated algorithms and entered the quality inspection process;

[0035] Calculate the percentage of data entries exempt from inspection to the total number of data entries;

[0036] The calculated percentage ratio is output as the exemption rate indicator in the evaluation report.

[0037] In some specific embodiments, the auxiliary processing cost of the automation algorithm is calculated based on the inspection item weight and the operation weight; the total manual processing cost is calculated based on the inspection item weight and the modification operation weight; and a cost-saving index is calculated based on the ratio of the auxiliary processing cost to the total manual processing cost, further including:

[0038] Iterate through all inspection items and calculate the product of the inspection item weight and the corresponding operation weight in the actual processing;

[0039] The product of all the inspection items is summed to obtain the auxiliary processing cost of the automated algorithm;

[0040] The total manual processing cost is calculated based on the weight of each inspection item and the weight of the modification operation.

[0041] By comparing the auxiliary processing costs with the costs of all-manual processing, a cost-saving index, which represents the degree of cost savings, is derived.

[0042] Based on the same concept, the present invention also provides a data annotation cost measurement system, comprising:

[0043] The processing flow scope determination module is configured to determine the scope of the digital processing flow to be evaluated, which clarifies the data format, data scale, annotation rules and quality requirements based on the annotation task description document;

[0044] The exemption standard setting module is configured to set exemption standards to determine whether the data output by the automated algorithm meets the quality requirements and is exempt from manual inspection. The exemption standard setting references the benchmark qualification standard and the pre-labeling results of the automated algorithm.

[0045] The weight allocation module is configured to assign inspection item weights to each inspection item within the scope of the digital processing flow, with the sum of the weights of all inspection items being a fixed value. At the same time, it assigns operation weights to manual operation types, wherein the operation weights are positively correlated with the comprehensive cost of the corresponding operation.

[0046] The exemption rate calculation module is configured to calculate the data exemption rate after processing by the automated algorithm based on the exemption standard, wherein the data exemption rate is determined based on the ratio of the number of exempted data entries to the total number of data entries;

[0047] The cost-saving index calculation module is configured to calculate the auxiliary processing cost of the automated algorithm based on the inspection item weight and operation weight, calculate the full manual processing cost based on the inspection item weight and modification operation weight, and calculate the cost-saving index according to the ratio of the auxiliary processing cost to the full manual processing cost.

[0048] Based on the same concept, the present invention also provides an electronic device, including: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of a data labeling cost measurement method.

[0049] Based on the same concept, the present invention also provides a computer-readable storage medium storing a computer program executable by an electronic device, which, when run on the electronic device, causes the electronic device to perform the steps of a data annotation cost measurement method.

[0050] Compared with existing technologies, its advantages are as follows:

[0051] This invention discloses a data annotation cost measurement method, system, electronic device, and storage medium, which visualizes cost changes: by introducing "inspection exemption rate" and "cost saving index," these are mapped to precisely measurable changes in labor costs. The percentage change in the evaluation results directly reflects the percentage change in labor costs, enabling managers to accurately quantify the economic contribution of the algorithm and providing direct and reliable data support for budget preparation and resource allocation.

[0052] The research and development direction has been clarified, providing optimization guidance for the algorithm development team. Data exempt from inspection requires no manual labor. Optimizing algorithm output reduces the difficulty and cost of correcting data requiring inspection (i.e., reducing the proportion of high-weight operations).

[0053] Efficient Management Decision-Making: It provides a unified quantitative benchmark for algorithm selection, investment decisions, and production line efficiency evaluation. By comparing the cost-saving indices of different algorithms or different versions of the same algorithm, managers can make decisions based on clear economic data, improving the scientific nature and efficiency of decision-making.

[0054] The measurement system is refined by introducing "inspection item weights" and "operation weights" to simulate the real cost structure of manual re-inspection. This weighting system matches the actual piece-rate wage calculation ratio, ensuring that the economic evaluation is consistent with the real cost structure. This allows for an accurate reflection of the differentiated impact of operations of varying complexity on the total cost, achieving a refined and precise measurement of the algorithm's economic value. Attached Figure Description

[0055] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0056] Figure 1 This is a flowchart illustrating some specific embodiments of the data annotation cost measurement method of the present invention;

[0057] Figure 2 This is one of the flowcharts of another embodiment of the data annotation cost measurement method of the present invention;

[0058] Figure 3 This is a second flowchart illustrating another embodiment of the data annotation cost measurement method of the present invention;

[0059] Figure 4 This is the third flowchart of another embodiment of the data annotation cost measurement method of the present invention;

[0060] Figure 5 This is the fourth flowchart of another embodiment of the data annotation cost measurement method of the present invention;

[0061] Figure 6 This is a schematic diagram of the structure of a data annotation cost measurement system according to some specific embodiments of the present invention;

[0062] Figure 7 This is a schematic diagram of the structure of an electronic device according to some specific embodiments of the present invention;

[0063] In the diagram, 710 is the processor; 720 is the memory; 730 is the input device; and 740 is the output device. Detailed Implementation

[0064] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0065] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The singular forms “a,” “said,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms, and “multiple” generally includes at least two unless the context clearly indicates otherwise.

[0066] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0067] It should be understood that although the terms first, second, third, etc., may be used in the embodiments of this application, these descriptions should not be limited to these terms. These terms are only used to distinguish the descriptions. For example, first may also be referred to as second without departing from the scope of the embodiments of this application, and similarly, second may also be referred to as first.

[0068] Depending on the context, the words “if” or “suppose” as used here can be interpreted as “when” or “in response to determination” or “in response to detection.” Similarly, depending on the context, the phrases “if determination” or “if detection (of the stated condition or event)” can be interpreted as “when determination” or “in response to determination” or “when detection (of the stated condition or event)” or “in response to detection (of the stated condition or event).”

[0069] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that an article or device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such an article or device. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the article or device that includes said element.

[0070] It should be noted that any symbols and / or numbers present in the specification that are not marked in the accompanying drawings are not reference numerals.

[0071] Reference Figure 1 A data annotation cost measurement method, comprising:

[0072] S101, Determine the scope of the digital processing flow to be evaluated, including clarifying the data format, data scale, annotation rules and quality requirements based on the annotation task description document;

[0073] S102, Establish inspection exemption standards to determine whether the data output by the automated algorithm meets the quality requirements and is exempt from manual inspection, wherein the establishment of the inspection exemption standards refers to the benchmark qualification standards and the pre-labeling results of the automated algorithm;

[0074] S103, assign inspection item weights to each inspection item within the scope of the digital processing flow, the sum of the weights of all inspection items is a fixed value, and at the same time assign operation weights to manual operation types, wherein the operation weights are positively correlated with the comprehensive cost of the corresponding operation.

[0075] S104, calculate the data exemption rate after processing by the automated algorithm based on the exemption standard, wherein the data exemption rate is determined based on the ratio of the number of exempted data entries to the total number of data entries;

[0076] S105, calculate the auxiliary processing cost of the automation algorithm based on the inspection item weight and operation weight, calculate the full manual processing cost based on the inspection item weight and modification operation weight, and calculate the cost saving index according to the ratio of the auxiliary processing cost to the full manual processing cost.

[0077] Specifically, in this embodiment of the invention, the scope of the digital processing flow to be evaluated is determined by analyzing the annotation task specification document provided by the data requester, which clarifies the specific form of the data, the estimated total amount of data, the annotation rules to be followed, and the quality requirements to be met. Exemption criteria are established to determine whether the output data of the automated algorithm is exempt from manual inspection. These criteria are based on baseline compliance standards that all data must meet, and refer to the performance results obtained by the automated algorithm through pre-annotation on representative sample datasets. Inspection item weights are assigned to each inspection item within the determined evaluation scope, ensuring that the sum of all weights is a fixed total value. This weight allocation can be based on the business importance of the inspection item or its historical piece-rate wage ratio. Operation weights are assigned to various types of manual operations that may be performed in the algorithm-assisted environment. The numerical value of the operation weight is positively correlated with the comprehensive cost factors such as time cost, skill requirements, and mental workload required to perform the operation. Based on the aforementioned exemption criteria, the data processed by the automated algorithm is statistically analyzed to calculate its exemption rate. The specific value of the exemption rate is determined by the proportion of data items granted exemption labels to the total number of data items processed by the algorithm. In the cost-benefit quantification stage, the cost of automated algorithm-assisted processing is obtained by multiplying the weight of each inspection item with the weight of the corresponding operation in the actual algorithm-assisted processing and summing the results. The cost of all manual processing is obtained by multiplying the weight of each inspection item with the weight of the modification operation and summing the results. Based on the ratio between the cost of automated algorithm-assisted processing and the cost of all manual processing, a cost-saving index, which represents the degree of cost savings, is calculated.

[0078] In some applications, the scope of the digitization process to be evaluated is determined. This involves clarifying the data format, data scale, annotation rules, and quality requirements based on the annotation task specification document. This includes parsing the annotation task specification document to extract data storage format, file type, and estimated total data volume information. Based on the annotation specifications in the annotation task specification document, the scope of inspection items to be evaluated is defined, forming a set of inspection items. The annotation rules, definitions of professional terms, and descriptions of various quality requirements specified in the annotation task specification document are determined. A quality indicator system is established, including accuracy, completeness, and consistency indicators, as well as corresponding acceptable threshold ranges.

[0079] Understandably, this involves analyzing the annotation task specification document provided by the data requester to extract key information, including the specific format of data storage, file type, and estimated total data volume. Based on the annotation specifications clearly stated in the annotation task specification document, a systematic definition of the inspection items to be included in the evaluation is established, forming a complete and clear set of inspection items. Furthermore, it is necessary to accurately understand and confirm the specific content of each annotation rule detailed in the annotation task specification document, the precise definitions of relevant professional terms, and the detailed descriptions of all quality requirements. A quality indicator system for acceptance is determined, which should at least include accuracy indicators, completeness indicators, and consistency indicators, and set clear and quantifiable pass / fail threshold ranges for each indicator, thereby completely defining the scope of the digitization process to be evaluated.

[0080] In some applications, exemption criteria are established to determine whether the data output by the automated algorithm meets quality requirements and is exempt from manual inspection. The exemption criteria are established with reference to benchmark acceptance standards and the pre-labeling results of the automated algorithm, including establishing benchmark acceptance standards for all data outputs; analyzing the pre-labeling results of the automated algorithm on the sample dataset, where the pre-labeling results include precision and recall metrics; combining the benchmark acceptance standards and the pre-labeling results to determine exemption condition parameters and judgment rules; and integrating the exemption condition parameters and judgment rules.

[0081] Understandably, a baseline qualification standard must be established that all data outputs must meet. This standard should be a quantifiable and verifiable quality threshold. The pre-labeling results of the automated algorithm on representative sample datasets should be analyzed, and the analysis of the pre-labeling results should include at least the evaluation of accuracy and recall metrics. Combining the baseline qualification standard and the analysis of the pre-labeling results, specific exemption condition parameters and corresponding judgment rules should be determined. These condition parameters and rules should be able to accurately distinguish data batches that meet the quality requirements. The determined exemption condition parameters and judgment rules should be integrated into the quality control module of the labeling system to realize an automated exemption judgment process.

[0082] In some applications, inspection item weights are assigned to each inspection item within the scope of the digital processing flow, and the sum of the weights of all inspection items is a fixed value. Simultaneously, operation weights are assigned to manual operation types, wherein the operation weights are positively correlated with the overall cost of the corresponding operation. This includes assigning corresponding inspection item weights based on the business importance and frequency of use of each inspection item in the data application scenario, or assigning inspection item weights based on the historical data proportion of each inspection item in the piece-rate wage accounting system. Normalization processing is used to ensure that the sum of the inspection item weights of all inspection items reaches the fixed value. The proportion of inspection item weight allocation is kept consistent with the billing proportion in the actual production cost accounting.

[0083] Understandably, the weights of each inspection item are allocated based on its importance and frequency of use in the final data application scenario, or based on its historical proportion in the piece-rate wage calculation system. Normalization is then used to ensure that the weights of all inspection items accumulate to a pre-set fixed value. Simultaneously, it is ensured that the weight allocation ratio of inspection items is consistent with the billing ratio in actual production cost accounting. Based on the completed weight allocation of inspection items, operation weights are assigned to manual operation types. The determination of operation weights needs to consider the comprehensive cost factors of the corresponding operation, including time cost, skill requirements, and operational complexity, so that the operation weight value is positively correlated with the comprehensive cost, ultimately establishing a complete weight allocation system.

[0084] In some applications, operation weights are assigned to manual operation types, wherein the operation weights are positively correlated with the overall cost of the corresponding operation. This includes defining a set of operation types performed by human operators in an automated algorithm-assisted environment; analyzing the time cost, skill requirements, and operational complexity factors required for each operation type; determining the relative weight values ​​of each operation type relative to the benchmark operation based on the comprehensive analysis results; and establishing a comparable operation weight system between different operation types through standardization.

[0085] Understandably, this involves defining all the types of operations that a human operator needs to perform in an automated algorithm-assisted environment, forming a complete set of operation types. The analysis then examines the influencing factors for each operation type, including time cost, skill requirements, and operational complexity. Time cost includes the average time to complete a single operation; skill requirements include the required knowledge background and training difficulty; and operational complexity includes the cumbersomeness of the operation steps and the difficulty of decision-making. Based on the comprehensive analysis results, the relative weight of each operation type relative to the benchmark operation is determined. The operation type with the lowest overall cost is set as the benchmark operation and assigned a benchmark weight value. Other operation types have their corresponding weights determined based on their relative cost coefficients. Finally, a comparable operation weight system is established through standardization to ensure that the weight values ​​accurately reflect the proportional relationship of each operation type in actual cost consumption.

[0086] In some applications, the data exemption rate after processing by the automated algorithm is calculated based on the exemption standard. The data exemption rate is determined based on the ratio of the number of exempted data entries to the total number of data entries. This includes obtaining the number of automatically exempted data entries within a set statistical time period; obtaining the total number of data entries processed by the automated algorithm and entering the quality inspection process within the same statistical time period; calculating the percentage ratio of the number of exempted data entries to the total number of data entries; and outputting the calculated percentage ratio as the exemption rate indicator to the evaluation report.

[0087] Understandably, within a set statistical time period, the number of data entries automatically determined to be exempt from inspection is retrieved from the database of the annotation system; within the same statistical time period, the total number of data entries processed by the automated algorithm and entering the quality inspection stage is retrieved; the percentage of exempt data entries in the total number of data entries is calculated by dividing the number of exempt data entries by the total number of data entries and then multiplying by 100%; the calculated percentage is output as the exemption rate indicator in the evaluation report for subsequent economic benefit analysis and algorithm performance evaluation.

[0088] In some applications, the auxiliary processing cost of the automation algorithm is calculated based on the inspection item weights and operation weights; the total manual processing cost is calculated based on the inspection item weights and modification operation weights; and a cost-saving index is calculated based on the ratio between the auxiliary processing cost and the total manual processing cost. This includes iterating through all inspection items and calculating the product of the inspection item weight and the corresponding operation weight in the actual processing; summing the product results of all inspection items to obtain the auxiliary processing cost of the automation algorithm; calculating the total manual processing cost based on the inspection item weights and modification operation weights of each inspection item; and obtaining a cost-saving index that characterizes the degree of cost savings by comparing the auxiliary processing cost and the total manual processing cost.

[0089] Understandably, when calculating the cost-saving index, all inspection items are iterated through, and the product of the inspection item weight and the corresponding operation weight in the actual processing is calculated. This product reflects the processing cost contribution of the inspection item with algorithm assistance. The product calculation results of all inspection items are summed to obtain the total processing cost assisted by the automated algorithm. This total value represents the relative cost required to complete the entire annotation task with algorithm assistance. The cost of full manual processing is calculated based on the inspection item weight and the modification operation weight. The total cost of full manual processing is obtained by multiplying the weight of each inspection item by the modification operation weight and summing the results. This total value represents the relative cost required to complete the same task purely manually without algorithm assistance. By comparing the auxiliary processing cost and the full manual processing cost, a specific mathematical formula is used to calculate the cost-saving index, which characterizes the degree of cost savings.

[0090] The following is combined Figures 2-5 Another embodiment of the data annotation cost measurement method of the present invention is described below:

[0091] In this embodiment:

[0092] like Figure 2 , Figure 3 and Figure 5As shown, defining the evaluation scope involves three parties: the data requester, the annotation manager, and the data annotator. The main process includes three stages: pre-annotation task preparation, annotation task execution, and annotation result output. Within this data annotation process framework, the annotation manager and the data annotator need to clarify the data format, data scale, annotation rules, terminology definitions, annotation examples, quality requirements, and acceptance criteria based on the annotation task description document provided by the data requester in the pre-annotation stage, thereby determining the evaluation scope.

[0093] Establishing exemption criteria: This is a set of quantitative standards and technical conditions used to determine whether batches of data output by automated algorithms meet preset quality requirements, thereby exempting them from manual quality inspection. It includes the following steps:

[0094] Establish baseline compliance standards: Collaborate with data requesters to confirm baseline compliance standards for data annotation. This standard represents the minimum quality threshold that all data, including manually annotated and automated algorithm-generated data, must meet. This standard must be quantifiable and verifiable. For regulated professional fields such as medicine, finance, and publishing, the establishment of these baseline compliance standards must comply with relevant national regulations and industry standards. For example, regulations may stipulate that the final annotation results must be sampled and verified by a qualified third-party organization, inviting experts with the appropriate qualifications and experience, to ensure compliance.

[0095] Establish inspection exemption criteria based on pre-labeled verification:

[0096] In the initial evaluation phase of the annotation task, the data manager or annotator uses the automated algorithm to pre-annotate a small-scale sample dataset provided by the data requester. Based on the pre-annotation results, the data requester and data manager jointly formulate specific exemption criteria. The criteria include, but are not limited to: quantitative indicators such as the accuracy and recall of the pre-annotation results relative to the baseline acceptance criteria; the model confidence accompanying the output results of the automated algorithm; historical accuracy statistics of the algorithm in similar tasks; and the criticality level of the current annotation task (e.g., the degree of security requirements).

[0097] System Configuration and Automated Execution: The established exemption criteria are configured in the labeling system. During subsequent large-scale automated labeling, the data output by the algorithm is monitored in real time, and batches of data that fully meet the exemption criteria are automatically labeled "exempt." Data tagged "exempt" will either directly enter the delivery process or undergo only a very low percentage of random sampling, thus achieving intelligent allocation of quality inspection resources.

[0098] Assign weights to each inspection item within the evaluation scope: The annotation management party needs to work with the data requester to set the weights of the inspection items based on their importance or piece-rate wage percentage, while also anonymizing them. The weight allocation typically requires the sum of the weights to be 1 or 100%. The importance of the inspection items is primarily determined by professionals familiar with the data usage scenario, assigning weight values ​​to each inspection item based on the application scenario of the data annotation results within the project context. The piece-rate wage percentage is determined by the annotation management party from a cost calculation and compensation allocation perspective, linking the annotators' remuneration to inspection items with different weights. Anonymization means that the metadata used when setting and calculating weights (such as annotator ID, specific salary amount, etc.) is processed and does not expose sensitive personal information. The system may only use anonymous employee IDs and corresponding weight scores for calculations, without directly linking to real names and bank accounts.

[0099] Assigning weights to different operation types: After adopting automated annotation algorithms, the core task of annotators becomes to review and correct the quality of the pre-annotated results generated by the algorithm.

[0100] Definition and Examples of Operation Types: Manual operation types refer to all interactive behaviors performed by annotators to correct output errors of automated algorithms or to confirm their results. These operations vary significantly in complexity, time consumption, and required skills, resulting in vastly different human resource costs. Taking a typical data annotation scenario as an example, the following basic operation types are defined and corresponding to typical algorithm errors:

[0101] Check: Abbreviated as "check," this is an operation performed by the labeling personnel to verify the labeling results output by the automated algorithm to confirm their correctness. This operation does not change the original output results of the algorithm.

[0102] Adding: Abbreviated as "adding", refers to the operation of supplementing annotations performed by annotation personnel to address omissions and errors in the automatic annotation algorithm.

[0103] Deletion: Abbreviated as "delete", refers to the operation of removing labels performed by labelers when there are multiple labeling errors in automated algorithms.

[0104] Modification: Abbreviated as "modification", refers to the operation of correcting label content (such as entity boundaries, classification categories, etc.) by labelers when there are mislabeling errors in automated algorithms.

[0105] Furthermore, setting an exemption rate allows some high-quality data to be directly adopted without manual processing. This type of operation is defined as exemption, or simply "exemption," and its corresponding labor cost is zero.

[0106] The logic for allocating operation weights is based on a core principle: the weight value should be positively correlated with the overall cost incurred by that operation type. This overall cost includes, but is not limited to, the operation's time complexity, mental workload, and skill requirements. To quantify this principle, a standardized weight allocation process is constructed, the core of which lies in converting the difficulty and time consumption of different operations relative to basic operations into quantifiable weights. This process can be further elaborated as follows:

[0107] Benchmark setting: The operation type with the lowest overall cost is set as the benchmark operation, and a benchmark weight value is assigned to it. For example, in data annotation, "inspection" is usually set as the base unit price representing "standard man-hours".

[0108] Relative Assessment: The system evaluates the relative cost coefficients of other operations relative to the baseline operation. This assessment can be based on historical time data, expert experience, or empirical analysis. For example, in data annotation, the operation with the lowest overall cost, "inspection," can be defined as the piece-rate baseline, and then the relative difficulty coefficients of other operations such as "add," "delete," and "modify" can be evaluated relative to it.

[0109] System normalization: The relative cost coefficients are normalized into a complete operational weighting system to ensure that the value ratio between different operational types is accurately reflected.

[0110] The underlying logic of the weight allocation is to convert operations of different natures into a unified measurable value unit through a normalized weight system, thereby achieving fair measurement.

[0111] Calculate the data exemption rate after algorithm processing: The exemption rate refers to the proportion of the total amount of data processed by the automated algorithm and assigned the "exempt" label by the system within a specific statistical period, relative to the total amount of data processed by the algorithm during the same period. The calculation formula is as follows:

[0112] ;

[0113] The percentage cost savings achieved by using automated annotation algorithms compared to the total cost of purely manual annotation. The calculation formula is as follows:

[0114] ;

[0115] In the algorithm evaluation phase, the actual algorithm-assisted processing cost is simulated. Based on the piece-rate wage calculation system, the weights of the inspection and modification operations are introduced after anonymization, and the cost is estimated through a weighted calculation method. The calculation formula is as follows:

[0116] ;

[0117] The maximum labor cost refers to the cost required to manually complete all data labeling item by item without the intervention of algorithms. Since this cost is difficult to obtain directly in an automated production line, it can be estimated using approximate methods.

[0118] Considering that modification operations have the highest weight in automated production lines, the cost of modifying all data items constitutes the maximum processing cost the system can potentially incur. Therefore, this maximum processing cost can be used to approximate the cost of all manual processing. The calculation formula is as follows:

[0119] ;

[0120] Output the comprehensive evaluation of the algorithm's economic efficiency: Based on the aforementioned steps, and assuming that the weights of the inspection items and operations have been set, the following process is executed to output the comprehensive evaluation:

[0121] The algorithm-assisted processing cost is calculated based on the number of inspection items within the evaluation scope.

[0122] Calculate the cost of all manual processing;

[0123] Substituting the above costs into the formula, the final cost-saving index is calculated, which directly represents the proportion of labor costs that can be saved by using the current algorithm version.

[0124] The following describes this embodiment in conjunction with application scenarios:

[0125] like Figure 2 , Figure 4 and Figure 5 As shown, the data is first pre-labeled by an automated annotation algorithm, and then manually checked and modified to meet the final data quality standards.

[0126] Determine the evaluation scope: Based on the annotation task description document provided by the data requester, the data format is determined to be TIFF and PDF files. For ease of explanation of the calculation rules, the data scale is tentatively set at 5 articles, and the quality requirement and acceptance criterion are an error rate not exceeding 0.02%. The evaluation scope of this embodiment is determined to be the annotation process of 10 metadata items for a single journal article, covering 10 metadata items of the journal academic paper. The check items are the annotation results of these 10 metadata items, as detailed in the metadata item table:

[0127]

[0128] Establish exemption criteria: These criteria determine whether the data output by the automated algorithm meets quality requirements without requiring manual inspection. In this embodiment, the automated algorithm pre-annotates the sample data provided by the client, and the measured error rate is 0.05%. However, given that the journal's metadata will be used for formal publication, its quality must be mandated to meet the national press and publication administration's requirement that the error rate not exceed 0.02% (i.e., accuracy ≥ 99.98%).

[0129] Assign weights to each metadata item within the evaluation scope: After determining that additional manual annotation is needed, the annotation management party must work with the data requester to set the weights of the check items based on the importance of the metadata items or the proportion of piece-rate wages, and the sum of the check item weights should be 1. The metadata annotated in this instance is primarily for database retrieval; therefore, professionals from the data requester who understand the data usage scenario will assign weight values ​​to each metadata item in the journal articles. See the journal article check item weight table for details.

[0130]

[0131] Weighting different operation types: After adding automated annotation algorithms, the work of annotators becomes a human-machine collaborative mode, mainly checking and correcting the pre-annotation results of the algorithms. To fairly measure the actual workload and value of annotators in the task of "checking and handling algorithm errors," we set weights for four operations—query, add, delete, and modify—based on piece-rate wages and anonymization. When assigning operation weights, a base unit price is first set, and then this base unit price is regarded as a "standard working hour" or "value benchmark." Finally, the difficulty and time consumption of the four operations—query, add, delete, and modify—relative to this benchmark are converted into weights. Therefore, the simplest operation, "check," is defined as the piece-rate benchmark, and the difficulty coefficients of other correction operations relative to it are evaluated. Finally, these coefficients are normalized into a weight system. Based on the pre-annotated data, the annotation management, after anonymization, linked the annotator's compensation to metadata items with different weights from the perspective of cost calculation and salary allocation, setting weight values ​​for the four types of operations. To fully describe the human-machine collaborative annotation work, exemption from inspection is also regarded as an operation type and a weight value is set. See the operation weight table for details.

[0132]

[0133] Calculate the data exemption rate after algorithm processing: The exemption rate refers to the proportion of data that does not require manual inspection after being processed by an automated algorithm to the total amount of data. The calculation formula is as follows:

[0134] ;

[0135] This annotation project covers 5 documents, requiring the annotation of 50 metadata items, resulting in a total of 50 data entries. The automated algorithm's pre-annotation results show that 45 metadata items are exempt from inspection. The exemption rate is calculated using the following formula:

[0136] =90%;

[0137] Therefore, the exemption rate for this project was 90%. This indicates that the project achieved a 90% exemption rate, which directly corresponds to saving approximately 90% of the direct manual inspection and modification workload in the data manual inspection stage.

[0138] Calculating the cost-saving index: Based on the previously set weights for inspection items, operation weights, total annotation volume, and exemption volume for journal literature metadata annotation, the cost-saving index can be calculated using a preset formula. To clearly present the parameters required for the calculation, the above data are summarized in the Journal Metadata Processing Operation Details Table:

[0139]

[0140] The cost-saving index is calculated below based on the formulas for calculating algorithm-assisted processing costs and fully manual processing costs, as well as the data provided in the table above. The process is as follows:

[0141] Algorithm-assisted processing cost: Since the title, Chinese abstract, and other 9 items are exempt from inspection, the algorithm-assisted processing cost for the inspection items is 0. Only the algorithm-assisted processing cost for the author's institution needs to be calculated. Author Affiliation: 2*0.3*1+1*0.3*2+1*0.3*2+1*0.3*4=3; Total Manual Processing Cost: Title: 5*0.2*4=4; Author Affiliation: 5*0.3*4=6; Chinese Abstract: 5*0.1*4=2; Chinese Keywords: 5*0.1*4=2; Classification Number: 5*0.05*4=1; References: 5*0.07*4=1.4; Funding: 5*0.06*4=1.2; Section Level: 5*0.05*4=1; DOI: 5*0.04*4=0.8; OA: 5*0.03*4=0.6; Total Manual Processing Cost = 4+6+2+2+1+1.4+1.2+1+0.8+0.6=20

[0142] Cost Savings Index: The cost savings index is calculated as follows:

[0143] =85%;

[0144] This means that using the current algorithm saves 85% of processing costs.

[0145] Output Algorithm Economic Evaluation: As mentioned earlier, the cost-saving index in the journal literature metadata annotation implementation case specifically refers to the percentage reduction in total cost when using automated annotation algorithms compared to purely manual annotation, under the premise of completing the same annotation task. A higher index indicates a greater role of automated algorithms in cost savings. In the current example of journal literature metadata annotation, the automated algorithm achieved a 90% exemption rate and a cost-saving index of 85%. This indicates that the exemption items covered by the automated algorithm have a higher overall weight, and the weight distribution of its exemption items is better than the average level of the overall projects.

[0146] For the purpose of simplicity, the method steps disclosed in the above embodiments are described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0147] like Figure 6 As shown, the present invention also provides a data annotation cost measurement system, comprising:

[0148] The processing flow scope determination module 201 is configured to determine the scope of the digital processing flow to be evaluated, wherein the data format, data scale, annotation rules and quality requirements are clearly defined based on the annotation task description document;

[0149] The inspection exemption standard setting module 202 is configured to set inspection exemption standards to determine whether the data output by the automated algorithm meets the quality requirements and is exempt from manual inspection. The inspection exemption standard setting references the benchmark qualification standard and the pre-labeling results of the automated algorithm.

[0150] The weight allocation module 203 is configured to assign inspection item weights to each inspection item within the scope of the digital processing flow, with the sum of the weights of all inspection items being a fixed value. At the same time, it assigns operation weights to manual operation types, wherein the operation weights are positively correlated with the comprehensive cost of the corresponding operation.

[0151] The exemption rate calculation module 204 is configured to calculate the data exemption rate after processing by the automated algorithm based on the exemption standard, wherein the data exemption rate is determined based on the ratio of the number of exempted data entries to the total number of data entries;

[0152] The cost-saving index calculation module 205 is configured to calculate the auxiliary processing cost of the automated algorithm based on the inspection item weight and operation weight, calculate the full manual processing cost based on the inspection item weight and modification operation weight, and calculate the cost-saving index according to the ratio of the auxiliary processing cost to the full manual processing cost.

[0153] It is worth noting that although only some basic functional modules are disclosed in the embodiments of this invention, it does not mean that the composition of this system is limited to the above-mentioned basic functional modules. On the contrary, what this embodiment intends to express is that, based on the above-mentioned basic functional modules, those skilled in the art can arbitrarily add one or more functional modules in combination with existing technology to form an infinite number of embodiments or technical solutions. That is to say, this system is open rather than closed. The fact that this embodiment only discloses a few basic functional modules should not be considered as the scope of protection of the claims of this invention being limited to the disclosed basic functional modules. At the same time, for the convenience of description, the above device is described separately according to its functions as various units and modules. Of course, in implementing this invention, the functions of each unit and module can be implemented in one or more software and / or hardware.

[0154] like Figure 7 As shown, the present invention also provides an electronic device, including: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of a data labeling cost measurement method.

[0155] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Figure 7 The structure shown in this embodiment of the invention includes an electronic device comprising one or more processors 710 and a memory 720; the processors 710 in this electronic device may be one or more. Figure 7 Taking a processor 710 as an example; memory 720 is used to store one or more programs; the one or more programs are executed by the one or more processors 710, so that the one or more processors 710 implement a data annotation cost measurement method as described in any one of the embodiments of the present invention.

[0156] The electronic device may also include an input device 730 and an output device 740.

[0157] The processor 710, memory 720, input device 730, and output device 740 in this electronic device can be connected via a bus or other means. Figure 7 Taking the example of a connection between China and Israel via a bus.

[0158] The memory 720 in this electronic device serves as a computer-readable storage medium, capable of storing one or more programs. These programs can be software programs, computer-executable programs, or modules, such as the program instructions / modules corresponding to the data annotation cost measurement method provided in this embodiment of the invention. The processor 710 executes various functional applications and data processing of the electronic device by running the software programs, instructions, and modules stored in the memory 720, thereby implementing the data annotation cost measurement method described in the above embodiment.

[0159] The memory 720 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device. Furthermore, the memory 720 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 720 may further include memory remotely located relative to the processor 710, which can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0160] Input device 730 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the electronic device. Output device 740 may include display devices such as a display screen.

[0161] The present invention also provides a computer-readable storage medium storing a computer program executable by an electronic device, which, when run on the electronic device, causes the electronic device to perform the steps of a data labeling cost measurement method.

[0162] Specifically, the computer storage medium in this embodiment of the invention can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. For example, a computer-readable storage medium can be—but is not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0163] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data annotation cost metric method, characterized by, The method comprises the following steps: determining the range of digital processing flow to be evaluated, wherein the data form, data size, labeling rules and quality requirements are specified based on the labeling task specification document; establishing an exemption standard for determining whether the data output by the automated algorithm meets the quality requirements and is exempt from manual inspection, wherein the establishment of the exemption standard refers to the reference qualified standard and the pre-labeling results of the automated algorithm; assigning a check item weight to each check item in the range of digital processing flow, and the sum of the weights of all check items is a fixed value, while assigning an operation weight to the manual operation type, wherein the operation weight is positively correlated with the comprehensive cost of the corresponding operation; calculating the data exemption rate after processing by the automated algorithm based on the exemption standard, wherein the data exemption rate is determined based on the ratio of the number of exempt data to the total number of data; calculating the auxiliary processing cost of the automated algorithm based on the check item weight and operation weight, calculating the full manual processing cost based on the check item weight and modification operation weight, and calculating the cost saving index according to the proportional relationship between the auxiliary processing cost and the full manual processing cost.

2. The method of claim 1, wherein, determining the range of digital processing flow to be evaluated, wherein the data form, data size, labeling rules and quality requirements are specified based on the labeling task specification document, further comprising: parsing the labeling task specification document to extract data storage format, file type and data total amount estimation information; defining the range of check items to be evaluated based on the labeling specification in the labeling task specification document to form a check item set; determining the labeling rule content, definition of professional terms and quality requirement specification specified in the labeling task specification document; determining the quality index system, including accuracy, completeness and consistency indicators and corresponding qualified threshold range.

3. The method of claim 1, wherein, establishing an exemption standard for determining whether the data output by the automated algorithm meets the quality requirements and is exempt from manual inspection, wherein the establishment of the exemption standard refers to the reference qualified standard and the pre-labeling results of the automated algorithm, further comprising: establishing the reference qualified standard for all data output; analyzing the pre-labeling results of the automated algorithm on the sample data set, wherein the pre-labeling results include accuracy and recall rate indicators; determining the exemption condition parameters and judgment rules based on the reference qualified standard and pre-labeling result analysis; integrating the exemption condition parameters and judgment rules.

4. The method of claim 1, wherein, assigning a check item weight to each check item in the range of digital processing flow, and the sum of the weights of all check items is a fixed value, while assigning an operation weight to the manual operation type, wherein the operation weight is positively correlated with the comprehensive cost of the corresponding operation, further comprising: assigning the corresponding check item weight according to the business importance and usage frequency of each check item in the data application scenario, or assigning the check item weight based on the historical data proportion relationship of each check item in the piecework wage accounting system; normalizing all check item weights to reach the fixed value after accumulation; keeping the check item weight allocation proportion consistent with the billing proportion in actual production cost accounting.

5. The method of claim 1, wherein, Assigning operation weights to the artificial operation types, wherein the operation weights are positively correlated with the comprehensive costs of the corresponding operations, and further comprising: Defining a set of operation types performed by artificial operators in an automated algorithm assisted environment; Analyzing the time cost, skill requirement and operation complexity influencing factors required for each operation type; Determining the relative weight values of each operation type relative to the benchmark operation according to the comprehensive analysis results; Establishing an operation weight system between different operation types through standardization processing to make them comparable.

6. The method of claim 1, wherein, Calculating the data exemption rate after the automated algorithm processing based on the exemption standard, wherein the data exemption rate is determined based on the ratio of the number of exempted data to the total number of data, and further comprising: Obtaining the number of data exempted from automatic determination within a set statistical time period; Obtaining the total number of data processed by the automated algorithm and entering the quality inspection link within the same statistical time period; Calculating the percentage of the number of exempted data in the total number of data; Outputting the calculated percentage ratio as the exemption rate index to the evaluation report.

7. The method of claim 1, wherein, Calculating the assisted processing cost of the automated algorithm based on the inspection item weight and operation weight, calculating the full artificial processing cost based on the inspection item weight and modified operation weight, and calculating the cost saving index according to the proportional relationship between the assisted processing cost and the full artificial processing cost, further comprising: Iterating through all inspection items and calculating the product of the inspection item weight of each inspection item and the corresponding operation weight in the actual processing process; Summing up the product calculation results of all inspection items to obtain the assisted processing cost of the automated algorithm; Calculating the full artificial processing cost based on the inspection item weight and modified operation weight of each inspection item; Obtaining the cost saving index representing the cost saving degree through the comparison operation of the assisted processing cost and the full artificial processing cost.

8. A data annotation cost metric system, comprising: Comprising: A processing flow range determination module configured to determine the digital processing flow range to be evaluated, wherein the data form, data size, labeling rules and quality requirements are specified based on the labeling task description document; An exemption standard development module configured to develop an exemption standard for determining whether the data output by the automated algorithm meets the quality requirements and is exempted from manual inspection, wherein the development of the exemption standard refers to the benchmark qualification standard and the pre-labeling results of the automated algorithm; A weight allocation module configured to allocate inspection item weights to each inspection item in the digital processing flow range, and the sum of the weights of all inspection items is a fixed value, and to allocate operation weights to artificial operation types, wherein the operation weights are positively correlated with the comprehensive costs of the corresponding operations; An exemption rate calculation module configured to calculate the data exemption rate after the automated algorithm processing based on the exemption standard, wherein the data exemption rate is determined based on the ratio of the number of exempted data to the total number of data; A cost saving index calculation module configured to calculate the assisted processing cost of the automated algorithm based on the inspection item weight and operation weight, calculate the full artificial processing cost based on the inspection item weight and modified operation weight, and calculate the cost saving index according to the proportional relationship between the assisted processing cost and the full artificial processing cost.

9. An electronic device, comprising: Comprising: A processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program executable by an electronic device, and when the computer program runs on the electronic device, the electronic device executes the steps of the method in any one of claims 1 to 7.

Images