Text labeling task allocation method and system based on artificial intelligence

Abstract

The embodiment of the invention discloses a text labeling task allocation method and system based on artificial intelligence, and the method comprises the steps: determining a work order type of a to-be-allocated work order through a labeling personnel determination model; judging whether Z1 exceeds a delivery time node Z required by the work order, and if yes, re-acquiring the to-be-distributed personnel of the work order until the completion time node of the to-be-distributed personnel of the work order does not exceed the delivery time node required by the work order; if not, storing the work order to-be-distributed personnel in a to-be-screened and distributed personnel pool, judging whether the line number of the to-be-distributed work orders is greater than a preset threshold N1, and if yes, selecting the work order to-be-distributed personnel with the highest labeling speed from the to-be-screened and distributed personnel pool; and judging whether the line number of the work order data is smaller than a preset threshold N2, and if so, selecting the work order to-be-distributed personnel with the highest annotation quality level from the to-be-screened distribution personnel pool. The problem that text labeling tasks cannot be distributed intelligently in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an artificial intelligence-based text annotation task allocation system, method, electronic device and storage medium. Background Art

[0002] Data labeling work orders refer to tasks or requests to process data such as images, voice, text, and video, mark the features of objects, and generate readable data codes that meet the requirements of machine learning training. This process involves converting raw data into a form that can be understood by machine learning algorithms. Through the labeled data, machines can learn various tasks such as classification, regression, and target detection. Currently, most data labeling work orders on most platforms are manually assigned. There are many data labelers on the platform, and there are many types of labeling work orders. Manually assigned work orders lead to unreasonable allocation, low work order completion quality, and work orders are prone to overdue.

[0003] Therefore, there is an urgent need for a method that can intelligently allocate text annotation tasks. Summary of the invention

[0004] The purpose of the embodiments of the present invention is to provide a text annotation task allocation system, method, electronic device and storage medium based on artificial intelligence, so as to solve the problem that text annotation task allocation cannot be performed intelligently in the prior art.

[0005] To achieve the above object, an embodiment of the present invention provides a method for allocating text annotation tasks based on artificial intelligence, and the method specifically includes:

[0006] Get the quality level and complexity of a certain number of historical work orders;

[0007] Based on the complexity of historical work orders and the time range required for completing the historical work orders as inputs of the work order marking time prediction model, the work order marking time prediction model is trained to obtain an optimal work order marking time prediction model;

[0008] Obtaining the work order to be assigned, calculating the work order to be assigned, and obtaining the complexity of the work order to be assigned;

[0009] Input the complexity of the work order to be assigned into the optimal work order marking time prediction model to obtain the expected time range [X, X1] for completing the work order to be assigned;

[0010] Constructing a labeling personnel determination model, determining the work order type of the work order to be assigned through the labeling personnel determination model, obtaining historical work orders corresponding to the work order type, obtaining data labeling personnel corresponding to the historical work orders, and selecting personnel to be assigned to the work order based on the data labeling personnel;

[0011] The completion time node of the work order to be assigned to the personnel is calculated by formula (1);

[0012] Z1=Z0+Y1+Y2+X1 formula (1);

[0013] In the formula, Z1 is the completion time node of the person to be assigned the work order, Z0 is the current time node, Y1 is the intermittent rest time between work orders within a certain time period for the person to be assigned the work order, Y2 is the sum of the upper limit time of the time range required for the work order corresponding to the current work order to be processed by the person to be assigned the work order, and X1 is the upper limit of the expected time range for the work order to be assigned to complete the work order;

[0014] Determine whether Z1 exceeds the delivery time node Z required by the work order. If so, re-acquire the personnel to be assigned to the work order until the completion time node of the personnel to be assigned to the work order does not exceed the delivery time node required by the work order;

[0015] If not, the personnel to be assigned to the work order is stored in a pool of personnel to be screened and assigned, and it is determined whether the number of work order rows to be assigned is greater than a preset threshold value N1. If so, the personnel to be assigned to the work order with the fastest marking speed is selected from the pool of personnel to be screened and assigned;

[0016] It is determined whether the number of work order data rows is less than a preset threshold value N2. If so, a work order to-be-assigned person with the highest marking quality level is selected from the pool of to-be-screened assigned persons.

[0017] On the basis of the above technical solution, the present invention can also be improved as follows:

[0018] Furthermore, obtaining the quality levels of a certain number of historical work orders includes:

[0019] The modified quality value of the work order is calculated by formula (2);

[0020]

[0021] The work order deletion quality value is calculated by formula (3);

[0022]

[0023] For different work order types, when (total number of work order lines - number of canceled lines) is greater than the line number threshold and the work order deletion quality value is greater than or equal to the first threshold and the work order modification quality value is greater than or equal to the second threshold, the work order quality level is level 1;

[0024] For different work order types, when (total number of work order lines - number of canceled lines) is greater than the line number threshold and the work order deletion quality value is greater than or equal to the first threshold or the work order modification quality value is greater than or equal to the second threshold, the work order quality level is level 2;

[0025] For different work order types, when (total number of work order lines - number of canceled lines) is greater than the line number threshold and the work order deletion quality value is less than the first threshold and the work order modification quality value is less than the second threshold, the work order quality level is level 3;

[0026] For different work order types, when the work order deletion quality value is equal to 0 and the work order modification quality value is less than the second threshold or the work order modification quality value is equal to 0 and the work order deletion quality value is less than the first threshold, the work order quality level is level 4;

[0027] For different work order types, when the work order deletion quality value is equal to 0 and the work order modification quality value is equal to 0, the work order quality level is level 5.

[0028] Further, the constructing of the labeling personnel determination model, determining the work order type of the work order to be assigned through the labeling personnel determination model, obtaining the historical work order corresponding to the work order type, obtaining the data labeling personnel corresponding to the historical work order, and selecting the personnel to be assigned to the work order based on the data labeling personnel, includes:

[0029] The labeling personnel determine the model, determine the work order type of the work order to be assigned, obtain the historical work order corresponding to the work order type, obtain the data labeling personnel corresponding to the historical work order, and determine the data labeling personnel with a quality level of 3 or above as the personnel to be assigned to the work order.

[0030] Further, the constructing of the labeling personnel determination model, determining the work order type of the work order to be assigned through the labeling personnel determination model, obtaining the historical work order corresponding to the work order type, obtaining the data labeling personnel corresponding to the historical work order, and selecting the personnel to be assigned to the work order based on the data labeling personnel, further includes:

[0031] Analyze the annotators through test questions and obtain their scores based on different dimensions;

[0032] Obtain historical data of all labelers on the platform, and predict the types of labeling work orders that the labelers are good at based on the historical data;

[0033] The best work order type for the first assignment of the labeler is obtained based on the labeler's scores based on different dimensions and the labeling work order types that the labeler is good at.

[0034] Further, the constructing of the labeling personnel determination model, determining the work order type of the work order to be assigned through the labeling personnel determination model, obtaining the historical work order corresponding to the work order type, obtaining the data labeling personnel corresponding to the historical work order, and selecting the personnel to be assigned to the work order based on the data labeling personnel, further includes:

[0035] During secondary allocation, different work order types with the same complexity level are allocated.

[0036] Furthermore, the complexity of obtaining a certain number of historical work orders includes:

[0037] Based on the field information to be extracted from the work order requirements, at least one triple is constructed, the relationship between the triples is analyzed, and based on the relationship between the triples and the number of triples, the entity relationship complexity of the work order to be assigned is obtained;

[0038] The text content complexity is calculated by formula (4);

[0039] Text content complexity = (number of sentences / number of characters)*100 Formula (4);

[0040] The amount of text content is calculated by formula (5);

[0041] Text content = number of text characters / 1000;

[0042] The complexity of the work order to be assigned is obtained through the entity relationship complexity, text content complexity and text content volume.

[0043] Further, the work order to-be-assigned personnel are stored in a pool of to-be-screened assigned personnel, and it is determined whether the number of work order rows to be assigned is greater than a preset threshold value N1, and if so, a work order to-be-assigned personnel with a fast marking speed is selected from the pool of to-be-screened assigned personnel; and it is determined whether the number of work order data rows is less than a preset threshold value N2, and if so, a work order to-be-assigned personnel with a high marking quality level is selected from the pool of to-be-screened assigned personnel, including:

[0044] When the number of work order rows is greater than the preset threshold value N1, the work order data of the work order type corresponding to the personnel to be assigned for each work order in the pool of personnel to be screened and assigned within a certain time period is obtained;

[0045] The average rate V of each work order type being processed by the personnel to be assigned to each work order is calculated by formula (6);

[0046] The average rate V of each work order type being processed by the personnel to be assigned to the work order = weighted average ((number of work order data rows * complexity of entity relationships of the work order * complexity of work order text content * amount of work order content) / actual time required to mark the work order) formula (6);

[0047] Select the person to be assigned the work order with the largest average rate V. If there are multiple people, select the person to be assigned the work order with the earliest completion time node;

[0048] When the number of work order rows is less than the preset threshold value N2, the work order data of the work order type corresponding to each work order to be assigned personnel in the pool of personnel to be screened and assigned within a certain period is obtained;

[0049] The average quality R of each work order type handled by the personnel to be assigned to each work order is calculated by formula (7);

[0050] Average quality R = weighted average (work order quality level) formula (7);

[0051] Select the person to be assigned the work order with the largest average quality R. If there are multiple people, select the person to be assigned the work order with the earliest completion time node.

[0052] A text annotation task allocation system based on artificial intelligence, comprising:

[0053] Text annotation quality analysis module, used to obtain the quality level of a certain number of historical work orders;

[0054] A work order complexity calculation module is used to obtain the complexity of a certain number of historical work orders, obtain the work orders to be assigned, calculate the work orders to be assigned, and obtain the complexity of the work orders to be assigned;

[0055] A text annotation time prediction module is used to train the work order annotation time prediction model based on the complexity of historical work orders and the time range required for the completion of historical work orders as inputs to obtain an optimal work order annotation time prediction model;

[0056] The labeling personnel feature testing module is used to build a labeling personnel determination model, determine the work order type of the work order to be assigned through the labeling personnel determination model, obtain the historical work order corresponding to the work order type, obtain the data labeling personnel corresponding to the historical work order, and select the personnel to be assigned to the work order based on the data labeling personnel;

[0057] A text annotation task assignment module is used to input the complexity of the work order to be assigned into the optimal work order annotation time prediction model to obtain the expected time range [X, X1] for completing the work order to be assigned;

[0058] The completion time node of the work order to be assigned to the personnel is calculated by formula (1);

[0059] Z1=Z0+Y1+Y2+X1 formula (1);

[0060] In the formula, Z1 is the completion time node of the person to be assigned the work order, Z0 is the current time node, Y1 is the intermittent rest time between work orders within a certain time period for the person to be assigned the work order, Y2 is the sum of the upper limit time of the time range required for the work order corresponding to the current work order to be processed by the person to be assigned the work order, and X1 is the upper limit time required for the work order to be assigned to complete the work order;

[0061] Determine whether Z1 exceeds the delivery time node Z required by the work order. If so, re-acquire the personnel to be assigned to the work order until the completion time node of the personnel to be assigned to the work order does not exceed the delivery time node required by the work order;

[0062] If not, store the personnel to be assigned to the work order into the pool of personnel to be screened and assigned, and determine whether the number of work order rows to be assigned is greater than the preset threshold value N1. If so, select the personnel to be assigned to the work order with a fast marking speed from the pool of personnel to be screened and assigned; determine whether the number of work order data rows is less than the preset threshold value N2. If so, select the personnel to be assigned to the work order with a high marking quality level from the pool of personnel to be screened and assigned.

[0063] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the steps of the method are implemented when the processor executes the computer program.

[0064] A non-transitory computer-readable storage medium stores a computer program, which implements the steps of the method when executed by a processor.

[0065] The embodiments of the present invention have the following advantages:

[0066] The text annotation task allocation method based on artificial intelligence in the present invention obtains the quality level and complexity of a certain number of historical work orders; based on the complexity of the historical work orders and the time range required for the completion of the historical work orders as the input of the work order annotation time prediction model, the work order annotation time prediction model is trained to obtain the optimal work order annotation time prediction model; the work order to be assigned is obtained, and the work order to be assigned is calculated to obtain the complexity of the work order to be assigned; the complexity of the work order to be assigned is input into the optimal work order annotation time prediction model to obtain the expected time range [X, X1] for completing the work order to be assigned; a labeling personnel determination model is constructed, and the work order type of the work order to be assigned is determined through the labeling personnel determination model, and the historical work order corresponding to the work order type is obtained, and the data corresponding to the historical work order is obtained. The labeling personnel select the personnel to be assigned to the work order based on the data labeling personnel; calculate the completion time node of the personnel to be assigned to the work order through a formula; determine whether Z1 exceeds the delivery time node Z required by the work order, if so, re-acquire the personnel to be assigned to the work order until the completion time node of the personnel to be assigned to the work order does not exceed the delivery time node required by the work order; if not, store the personnel to be assigned to the work order in the pool of personnel to be screened and assigned, and determine whether the number of work order rows to be assigned is greater than a preset threshold value N1, if so, select the personnel to be assigned to the work order with the fastest labeling speed from the pool of personnel to be screened and assigned; determine whether the number of work order data rows is less than a preset threshold value N2, if so, select the personnel to be assigned to the work order with the highest labeling quality level from the pool of personnel to be screened and assigned, which solves the problem that the existing technology cannot intelligently assign text labeling tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0068] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0069] Figure 1 A flowchart of a method for allocating text annotation tasks based on artificial intelligence according to the present invention;

[0070] Figure 2 This is an architecture diagram of the text annotation task allocation system based on artificial intelligence of the present invention;

[0071] Figure 3 This is a schematic diagram of the physical structure of the electronic device provided by the present invention.

[0072] The accompanying drawings are denoted as follows:

[0073] Text annotation quality analysis module 10, work order complexity calculation module 20, text annotation time prediction module 30, annotation personnel feature testing module 40, text annotation task allocation module 50, electronic device 60, processor 601, memory 602, bus 603. DETAILED DESCRIPTION

[0074] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0075] Figure 1 Flow chart of an embodiment of the method for allocating text annotation tasks based on artificial intelligence of the present invention, as shown in FIG. Figure 1 As shown, an artificial intelligence-based text annotation task allocation method provided by an embodiment of the present invention includes the following steps:

[0076] S101, obtain the quality level and complexity of a certain number of historical work orders;

[0077] Specifically, based on the data cleaning after data annotation, the degree of modification of the data annotation results caused by data cleaning is obtained. For example, for a field in a row, the ratio of the number of characters added or deleted during cleaning to the original number of characters in the field is the modification degree of the field in the current row. The modification degree value of the row is obtained by dividing the sum of the modification degrees of all fields in the row by the total number of fields, and the modification degree values ​​of all modified rows are counted.

[0078] The modified quality value of the work order is calculated by formula (2);

[0079]

[0080] The work order deletion quality value is calculated by formula (3);

[0081]

[0082] For different work order types, when (total number of work order lines - number of canceled lines) is greater than the line number threshold and the work order deletion quality value is greater than or equal to the first threshold and the work order modification quality value is greater than or equal to the second threshold, the work order quality level is level 1;

[0083] For different work order types, when (total number of work order lines - number of canceled lines) is greater than the line number threshold and the work order deletion quality value is greater than or equal to the first threshold or the work order modification quality value is greater than or equal to the second threshold, the work order quality level is level 2;

[0084] For different work order types, when (total number of work order lines - number of canceled lines) is greater than the line number threshold and the work order deletion quality value is less than the first threshold and the work order modification quality value is less than the second threshold, the work order quality level is level 3;

[0085] For different work order types, when the work order deletion quality value is equal to 0 and the work order modification quality value is less than the second threshold or the work order modification quality value is equal to 0 and the work order deletion quality value is less than the first threshold, the work order quality level is level 4;

[0086] For different work order types, when the work order deletion quality value is equal to 0 and the work order modification quality value is equal to 0, the work order quality level is level 5.

[0087] The complexity of obtaining a certain number of historical work orders includes:

[0088] Based on the field information to be extracted from the work order requirements, at least one triple is constructed, the relationship between the triples is analyzed, and based on the relationship between the triples and the number of triples, the entity relationship complexity of the work order to be assigned is obtained;

[0089] The text content complexity is calculated by formula (4);

[0090] Text content complexity = (number of sentences / number of characters)*100 Formula (4);

[0091] (0~0.1 (including 0.1) is level 1; 0.1~0.2 (including 0.2) is level 2...0.9~1 (including 0.2) is level 10);

[0092] The amount of text content is calculated by formula (5);

[0093] Text content = number of text characters / 1000 (round down);

[0094] The complexity of the work order to be assigned is obtained through the entity relationship complexity, text content complexity and text content volume.

[0095] For each enterprise's data annotation work order requirement on the platform, an expected completion time will be set. When assigning work orders, the delivery time of data annotation needs to be considered. When a work order is received, the complexity of the work order is analyzed to obtain three indicators: entity relationship complexity, text content complexity and content volume, as well as the delivery time node Z required by the work order; obtain the time range required for historical work orders: define the range in 5-minute classes according to the time required to complete historical work orders;

[0096] S102, based on the complexity of historical work orders and the time range required for completing the historical work orders as inputs of the work order marking time prediction model, the work order marking time prediction model is trained to obtain an optimal work order marking time prediction model;

[0097] Specifically, the complexity of historical work orders and the time range required to complete historical work orders are divided into training sets, validation sets, and test sets;

[0098] Training the work order annotation time prediction model based on the training set;

[0099] Performing a performance evaluation on the trained work order annotation time prediction model based on the validation set to obtain a work order annotation time prediction model that meets the performance requirements;

[0100] The prediction result of the work order marking time prediction model that meets the performance condition is evaluated based on the test set to obtain an evaluation index corresponding to the work order marking time prediction model.

[0101] Based on the validation set, a performance evaluation is performed on the trained work order annotation time prediction model to obtain a work order annotation time prediction model that meets the performance conditions; based on the test set, a similarity calculation result of the work order annotation time prediction model that meets the performance conditions is evaluated to obtain an evaluation index corresponding to the work order annotation time prediction model. A performance evaluation is performed on the work order annotation time prediction model to obtain a percentage score (i.e., the highest score is 100 points and the lowest score is 0 points), and a work order annotation time prediction model with a score greater than a set value is determined based on the percentage score. For example, a work order annotation time prediction model with a score greater than 90 points is a work order annotation time prediction model that meets the performance conditions;

[0102] An evaluation index is calculated for the work order marking time prediction model that meets the performance conditions to obtain the evaluation index of the work order marking time prediction model, and an evaluation value corresponding to each evaluation index is calculated to obtain the evaluation value, which is used to represent the capability value of the work order marking time prediction model based on the evaluation index.

[0103] S103, obtaining the work orders to be assigned, calculating the work orders to be assigned, and obtaining the complexity of the work orders to be assigned;

[0104] S104, input the complexity of the work order to be assigned into the optimal work order marking time prediction model to obtain the expected time range [X, X1] for completing the work order to be assigned;

[0105] S105, constructing a labeling personnel determination model, determining the work order type of the work order to be assigned through the labeling personnel determination model, obtaining historical work orders corresponding to the work order type, obtaining data labeling personnel corresponding to the historical work orders, and selecting personnel to be assigned to the work order based on the data labeling personnel;

[0106] Specifically, the model is determined by the labeling personnel, the work order type of the work order to be assigned is determined, the historical work orders corresponding to the work order type are obtained, the data labeling personnel corresponding to the historical work orders are obtained, and the data labeling personnel with a quality level of 3 or above are determined as the personnel to be assigned the work orders.

[0107] Analyze the labelers through test questions to get their scores based on different dimensions; when labelers join the labeling platform for the first time, analyze the labelers through pre-set test questions to understand their scores based on different dimensions. Predict the types of labeling orders that labelers are good at based on the historical data of all labelers on the platform, and build a labeler determination model, which mainly includes: obtaining the historical labelers of the labeler determination model training data, the scores of different dimensions generated by the labelers' test questions, and the highest quality work order types for the labelers' first labeling of different types of data. Based on the labeler determination model training, obtain the best work order type for the labeler's first assignment.

[0108] Obtain historical data of all labelers on the platform, and predict the types of labeling work orders that the labelers are good at based on the historical data;

[0109] Historical data includes: data on test questions for first-time users, types of work orders with historical data annotation, quality of completed work orders, etc.

[0110] The best work order type for the first assignment of the labeler is obtained based on the labeler's scores based on different dimensions and the labeling work order types that the labeler is good at.

[0111] In order to better tap the potential of labeling personnel, during the secondary allocation, work order types with the same or different complexity levels are allocated to avoid too single work order allocation, which will not make good use of labeling personnel resources.

[0112] S106, calculating the completion time node of the personnel to be assigned to the work order through a formula;

[0113] Specifically, the completion time node of the personnel to be assigned to the work order is calculated by formula (1);

[0114] Z1=Z0+Y1+Y2+X1 formula (1);

[0115] In the formula, Z1 is the completion time node of the person to be assigned the work order, Z0 is the current time node, Y1 is the intermittent rest time between work orders within a certain time period for the person to be assigned the work order, Y2 is the sum of the upper limit time of the time range required for the work order corresponding to the current work order to be processed by the person to be assigned the work order, and X1 is the upper limit of the expected time range for the work order to be assigned to complete the work order.

[0116] S107, determining whether Z1 exceeds the delivery time node Z required by the work order, if so, reacquiring the personnel to be assigned to the work order until the completion time node of the personnel to be assigned to the work order does not exceed the delivery time node required by the work order;

[0117] S108, if not, the personnel to be assigned to the work order are stored in the pool of personnel to be screened and assigned, and it is determined whether the number of work order rows to be assigned is greater than a preset threshold value N1. If so, the personnel to be assigned to the work order with the fastest marking speed is selected from the pool of personnel to be screened and assigned;

[0118] S109, determining whether the number of work order data rows is less than a preset threshold value N2, and if so, selecting the work order to-be-assigned personnel with the highest annotated quality level from the pool of personnel to be screened and assigned;

[0119] Specifically, when the number of work order rows is greater than a preset threshold value N1, the work order data of the work order type corresponding to the personnel to be assigned for each work order in the pool of personnel to be screened and assigned within a certain time period is obtained;

[0120] The average rate V of each work order type being processed by the personnel to be assigned to each work order is calculated by formula (6);

[0121] The average rate V of each work order type being processed by the personnel to be assigned to the work order = weighted average ((number of work order data rows * complexity of entity relationships of the work order * complexity of work order text content * amount of work order content) / actual time required to mark the work order) formula (6);

[0122] Select the person to be assigned the work order with the largest average rate V. If there are multiple people, select the person to be assigned the work order with the earliest completion time node;

[0123] When the number of work order rows is less than the preset threshold value N2, the work order data of the work order type corresponding to each work order to be assigned personnel in the pool of personnel to be screened and assigned within a certain period is obtained;

[0124] The average quality R of each work order type handled by the personnel to be assigned to each work order is calculated by formula (7);

[0125] Average quality R = weighted average (work order quality level) formula (7);

[0126] Select the person to be assigned the work order with the largest average quality R. If there are multiple people, select the person to be assigned the work order with the earliest completion time node.

[0127] A preferred embodiment is as follows:

[0128] S1: Data labelers at the same quality level, obtain the number of work orders a based on this work order type with a quality level of 3 or above, obtain the number of work orders b based on this work order type, c = a / b; give priority to the personnel with the larger work order c;

[0129] S2: Based on the work order to be assigned to the person M selected in S1, obtain the intermittent rest time Y1 between the work orders of M in the past week, obtain the sum of the upper limit time Y2 of the time range required for the work orders corresponding to the work orders currently to be processed by M, calculate according to the normal working time interval of each working day pre-set by the platform, calculate Y1+Y2+X1, judge based on the current time node plus Y1+Y2+X1, get the completion time node Z1, and judge whether Z1 exceeds the delivery time node Z required by the work order; if exceeded, return to S1; if not, enter S3:

[0130] S3: Store M in S2 into the pool of personnel to be screened and assigned according to the quality level of S1, and return to S1; until there are no personnel to be assigned for work orders that meet the condition (1) in S1, then enter S4;

[0131] Based on the principle that the number of work order data rows is large (the number of data rows is greater than the preset threshold N1, which can provide sufficient training data), the personnel with fast annotation speed are given priority, and the number of work order data rows is small (the number of data rows is less than the preset threshold N2), the personnel with high annotation quality level are given priority for assignment;

[0132] S4: When the number of work order rows is greater than N1, take the work order data of this work order type corresponding to each labeler in the pool of personnel to be screened and assigned in S3 in the past month. For each work order data, calculate the average rate V of each labeler processing this type of work order = weighted average ((number of work order data rows * complexity of entity relationship of work order * complexity of work order text content * amount of work order content) / actual time required to label work orders), select the labeler with the largest V, if there are multiple people, select the one with the earliest Z1; when the number of work order rows is less than N2, take the work order data of this work order type corresponding to each labeler in the pool of personnel to be screened and assigned in S3 in the past month, for each work order data, calculate the average quality R of each labeler processing this type of work order = weighted average (work order quality level), select the labeler with the largest R value, if there are multiple people, select the one with the earliest Z1.

[0133] The text annotation task assignment method based on artificial intelligence obtains the quality level and complexity of a certain number of historical work orders; based on the complexity of the historical work orders and the time range required for the completion of the historical work orders as the input of the work order annotation time prediction model, the work order annotation time prediction model is trained to obtain the optimal work order annotation time prediction model; the work order to be assigned is obtained, and the complexity of the work order to be assigned is calculated to obtain the complexity of the work order to be assigned; the complexity of the work order to be assigned is input into the optimal work order annotation time prediction model to obtain the expected time range [X, X1] for completing the work order to be assigned; a labeling personnel determination model is constructed, and the work order type of the work order to be assigned is determined through the labeling personnel determination model, and the historical work order corresponding to the work order type is obtained. The data annotator corresponding to the historical work order selects the personnel to be assigned to the work order based on the data annotator; calculates the completion time node of the personnel to be assigned to the work order through a formula; determines whether Z1 exceeds the delivery time node Z required by the work order, if so, re-acquires the personnel to be assigned to the work order until the completion time node of the personnel to be assigned to the work order does not exceed the delivery time node required by the work order; if not, stores the personnel to be assigned to the work order in the pool of personnel to be screened and assigned, determines whether the number of rows of work orders to be assigned is greater than a preset threshold value N1, if so, selects the personnel to be assigned to the work order with the fastest annotation speed from the pool of personnel to be screened and assigned; determines whether the number of rows of work order data is less than a preset threshold value N2, if so, selects the personnel to be assigned to the work order with the highest annotation quality level from the pool of personnel to be screened and assigned. The problem that the existing technology cannot intelligently assign text annotation tasks is solved.

[0134] Figure 2 FIG. 1 is a flow chart of an embodiment of a text annotation task allocation system based on artificial intelligence of the present invention; Figure 2 As shown, an embodiment of the present invention provides an artificial intelligence-based text annotation task allocation system, comprising the following steps:

[0135] The text annotation quality analysis module 10 is used to obtain the quality level of a certain number of historical work orders;

[0136] The text annotation quality analysis module 10 is also used for:

[0137] The modified quality value of the work order is calculated by formula (2);

[0138]

[0139] The work order deletion quality value is calculated by formula (3);

[0140]

[0141] For different work order types, when (total number of work order lines - number of canceled lines) is greater than the line number threshold and the work order deletion quality value is greater than or equal to the first threshold and the work order modification quality value is greater than or equal to the second threshold, the work order quality level is level 1;

[0142] For different work order types, when (total number of work order lines - number of canceled lines) is greater than the line number threshold and the work order deletion quality value is greater than or equal to the first threshold or the work order modification quality value is greater than or equal to the second threshold, the work order quality level is level 2;

[0143] For different work order types, when (total number of work order lines - number of canceled lines) is greater than the line number threshold and the work order deletion quality value is less than the first threshold and the work order modification quality value is less than the second threshold, the work order quality level is level 3;

[0144] For different work order types, when the work order deletion quality value is equal to 0 and the work order modification quality value is less than the second threshold or the work order modification quality value is equal to 0 and the work order deletion quality value is less than the first threshold, the work order quality level is level 4;

[0145] For different work order types, when the work order deletion quality value is equal to 0 and the work order modification quality value is equal to 0, the work order quality level is level 5.

[0146] The work order complexity calculation module 20 is used to obtain the complexity of a certain number of historical work orders, obtain the work orders to be assigned, calculate the work orders to be assigned, and obtain the complexity of the work orders to be assigned;

[0147] Based on the field information to be extracted from the work order requirements, at least one triple is constructed, the relationship between the triples is analyzed, and based on the relationship between the triples and the number of triples, the entity relationship complexity of the work order to be assigned is obtained; in addition, the relationship between these triples is analyzed, if the entity 1 or entity 2 of a triple is the same as the entity 1 or entity 2 of another triple, the complexity is increased by 0.5, and the sum of the complexity values ​​of the current work order is the entity relationship complexity of the work order obtained by analysis;

[0148] The text content complexity is calculated by formula (4);

[0149] Text content complexity = (number of sentences / number of characters)*100 Formula (4);

[0150] The amount of text content is calculated by formula (5);

[0151] Text content = number of text characters / 1000;

[0152] The complexity of the work order to be assigned is obtained through the entity relationship complexity, text content complexity and text content volume.

[0153] A text annotation time prediction module 30 is used to train the work order annotation time prediction model based on the complexity of the historical work orders and the time range required for completing the historical work orders as inputs to obtain an optimal work order annotation time prediction model;

[0154] The labeling personnel feature testing module 40 is used to build a labeling personnel determination model, determine the work order type of the work order to be assigned through the labeling personnel determination model, obtain the historical work order corresponding to the work order type, obtain the data labeling personnel corresponding to the historical work order, and select the personnel to be assigned to the work order based on the data labeling personnel;

[0155] The labeling personnel feature testing module 40 is also used for:

[0156] The labeling personnel determine the model, determine the work order type of the work order to be assigned, obtain the historical work order corresponding to the work order type, obtain the data labeling personnel corresponding to the historical work order, and determine the data labeling personnel with a quality level of 3 or above as the personnel to be assigned to the work order.

[0157] Analyze the annotators through test questions and obtain their scores based on different dimensions;

[0158] Obtain historical data of all labelers on the platform, and predict the types of labeling work orders that the labelers are good at based on the historical data;

[0159] The best work order type for the first assignment of the labeler is obtained based on the labeler's scores based on different dimensions and the labeling work order types that the labeler is good at.

[0160] During secondary allocation, different work order types with the same complexity level are allocated.

[0161] The text annotation task assignment module 50 is used to input the complexity of the work order to be assigned into the optimal work order annotation time prediction model to obtain the expected time range [X, X1] for completing the work order to be assigned;

[0162] The completion time node of the work order to be assigned to the personnel is calculated by formula (1);

[0163] Z1=Z0+Y1+Y2+X1 formula (1);

[0164] In the formula, Z1 is the completion time node of the person to be assigned the work order, Z0 is the current time node, Y1 is the intermittent rest time between work orders within a certain time period for the person to be assigned the work order, Y2 is the sum of the upper limit time of the time range required for the work order corresponding to the current work order to be processed by the person to be assigned the work order, and X1 is the upper limit time required for the work order to be assigned to complete the work order;

[0165] Determine whether Z1 exceeds the delivery time node Z required by the work order. If so, re-acquire the personnel to be assigned to the work order until the completion time node of the personnel to be assigned to the work order does not exceed the delivery time node required by the work order;

[0166] If not, store the personnel to be assigned to the work order in the pool of personnel to be screened, and determine whether the number of work order rows to be assigned is greater than a preset threshold value N1. If so, select the personnel to be assigned to the work order with a fast marking speed from the pool of personnel to be screened; determine whether the number of work order data rows is less than a preset threshold value N2. If so, select the personnel to be assigned to the work order with a high marking quality level from the pool of personnel to be screened;

[0167] When the number of work order rows is greater than the preset threshold value N1, the work order data of the work order type corresponding to the personnel to be assigned for each work order in the pool of personnel to be screened and assigned within a certain time period is obtained;

[0168] The average rate V of each work order type being processed by the personnel to be assigned to each work order is calculated by formula (6);

[0169] The average rate V of each work order type being processed by the personnel to be assigned to the work order = weighted average ((number of work order data rows * complexity of entity relationships of the work order * complexity of work order text content * amount of work order content) / actual time required to mark the work order) formula (6);

[0170] Select the person to be assigned the work order with the largest average rate V. If there are multiple people, select the person to be assigned the work order with the earliest completion time node;

[0171] When the number of work order rows is less than the preset threshold value N2, the work order data of the work order type corresponding to each work order to be assigned personnel in the pool of personnel to be screened and assigned within a certain period is obtained;

[0172] The average quality R of each work order type handled by the personnel to be assigned to each work order is calculated by formula (7);

[0173] Average quality R = weighted average (work order quality level) formula (7);

[0174] Select the person to be assigned the work order with the largest average quality R. If there are multiple people, select the person to be assigned the work order with the earliest completion time node.

[0175] Figure 3 A schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as Figure 3 As shown, the electronic device 60 includes: a processor 601 (processor), a memory 602 (memory) and a bus 603;

[0176] The processor 601 and the memory 602 communicate with each other via the bus 603;

[0177] The processor 601 is used to call the program instructions in the memory 602 to execute the methods provided by the above-mentioned method embodiments, for example, including: obtaining the quality level and complexity of a certain number of historical work orders; based on the complexity of the historical work orders and the time range required for the completion of the historical work orders as the input of the work order annotation time prediction model, training the work order annotation time prediction model to obtain the optimal work order annotation time prediction model; obtaining the work order to be assigned, calculating the work order to be assigned, and obtaining the complexity of the work order to be assigned; inputting the complexity of the work order to be assigned into the optimal work order annotation time prediction model to obtain the expected time range [X, X1] for completing the work order to be assigned; constructing a labeling personnel determination model, determining the work order type of the work order to be assigned through the labeling personnel determination model, and obtaining the Obtain the data labeling personnel corresponding to the historical work orders corresponding to the work order type, and select the personnel to be assigned to the work order based on the data labeling personnel; calculate the completion time node of the personnel to be assigned to the work order through a formula; determine whether Z1 exceeds the delivery time node Z required by the work order, if so, re-obtain the personnel to be assigned to the work order until the completion time node of the personnel to be assigned to the work order does not exceed the delivery time node required by the work order; if not, store the personnel to be assigned to the work order in the pool of personnel to be screened and assigned, and determine whether the number of work order rows to be assigned is greater than the preset threshold value N1, if so, select the personnel to be assigned to the work order with the fastest labeling speed from the pool of personnel to be screened and assigned; determine whether the number of work order data rows is less than the preset threshold value N2, if so, select the personnel to be assigned to the work order with the highest labeling quality level from the pool of personnel to be screened and assigned.

[0178] The present embodiment provides a non-transitory computer-readable storage medium, which stores computer instructions. The computer instructions enable a computer to execute the methods provided by the above-mentioned method embodiments, for example, including: obtaining the quality level and complexity of a certain number of historical work orders; training the work order annotation time prediction model based on the complexity of the historical work orders and the time range required for the completion of the historical work orders as inputs to the work order annotation time prediction model to obtain an optimal work order annotation time prediction model; obtaining the work order to be assigned, calculating the work order to be assigned, and obtaining the complexity of the work order to be assigned; inputting the complexity of the work order to be assigned into the optimal work order annotation time prediction model to obtain the expected time range [X, X1] for completing the work order to be assigned; constructing a labeling personnel determination model, and determining the work order to be assigned through the labeling personnel determination model. Match the work order type of the work order, obtain the historical work orders corresponding to the work order type, obtain the data annotation personnel corresponding to the historical work orders, and select the personnel to be assigned to the work order based on the data annotation personnel; calculate the completion time node of the personnel to be assigned to the work order through a formula; determine whether Z1 exceeds the delivery time node Z required by the work order, if so, re-obtain the personnel to be assigned to the work order until the completion time node of the personnel to be assigned to the work order does not exceed the delivery time node required by the work order; if not, store the personnel to be assigned to the work order in the pool of personnel to be screened and assigned, determine whether the number of work order rows to be assigned is greater than a preset threshold value N1, if so, select the personnel to be assigned to the work order with the fastest annotation speed from the pool of personnel to be screened and assigned; determine whether the number of work order data rows is less than a preset threshold value N2, if so, select the personnel to be assigned to the work order with the highest annotation quality level from the pool of personnel to be screened and assigned.

[0179] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various storage media that can store program codes.

[0180] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0181] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0182] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A method for allocating text annotation tasks based on artificial intelligence, characterized in that: The text annotation task allocation method based on artificial intelligence specifically includes: Get the quality level and complexity of a certain number of historical work orders; Based on the complexity of historical work orders and the time range required for completing the historical work orders as inputs of the work order marking time prediction model, the work order marking time prediction model is trained to obtain an optimal work order marking time prediction model; Obtaining the work order to be assigned, calculating the work order to be assigned, and obtaining the complexity of the work order to be assigned; Input the complexity of the work order to be assigned into the optimal work order marking time prediction model to obtain the expected time range [X, X1] for completing the work order to be assigned; Constructing a labeling personnel determination model, determining the work order type of the work order to be assigned through the labeling personnel determination model, obtaining historical work orders corresponding to the work order type, obtaining data labeling personnel corresponding to the historical work orders, and selecting personnel to be assigned to the work order based on the data labeling personnel; The completion time node of the work order to be assigned to the personnel is calculated by formula (1); Z1=Z0+Y1+Y2+X1 formula (1); In the formula, Z1 is the completion time node of the person to be assigned the work order, Z0 is the current time node, Y1 is the intermittent rest time between work orders within a certain time period for the person to be assigned the work order, Y2 is the sum of the upper limit time of the time range required for the work order corresponding to the current work order to be processed by the person to be assigned the work order, and X1 is the upper limit of the expected time range for the work order to be assigned to complete the work order; Determine whether Z1 exceeds the delivery time node Z required by the work order. If so, re-acquire the personnel to be assigned to the work order until the completion time node of the personnel to be assigned to the work order does not exceed the delivery time node required by the work order; If not, the personnel to be assigned to the work order is stored in a pool of personnel to be screened and assigned, and it is determined whether the number of work order rows to be assigned is greater than a preset threshold value N1. If so, the personnel to be assigned to the work order with the fastest marking speed is selected from the pool of personnel to be screened and assigned; It is determined whether the number of work order data rows is less than a preset threshold value N2. If so, a work order to-be-assigned person with the highest marking quality level is selected from the pool of to-be-screened assigned persons.

2. The method for allocating text annotation tasks based on artificial intelligence according to claim 1, characterized in that: The quality level of a certain number of historical work orders is obtained, including: The modified quality value of the work order is calculated by formula (2); The work order deletion quality value is calculated by formula (3); For different work order types, when (total number of work order lines - number of canceled lines) is greater than the line number threshold and the work order deletion quality value is greater than or equal to the first threshold and the work order modification quality value is greater than or equal to the second threshold, the work order quality level is level 1; For different work order types, when (total number of work order lines - number of canceled lines) is greater than the line number threshold and the work order deletion quality value is greater than or equal to the first threshold or the work order modification quality value is greater than or equal to the second threshold, the work order quality level is level 2; For different work order types, when (total number of work order lines - number of canceled lines) is greater than the line number threshold and the work order deletion quality value is less than the first threshold and the work order modification quality value is less than the second threshold, the work order quality level is level 3; For different work order types, when the work order deletion quality value is equal to 0 and the work order modification quality value is less than the second threshold or the work order modification quality value is equal to 0 and the work order deletion quality value is less than the first threshold, the work order quality level is level 4; For different work order types, when the work order deletion quality value is equal to 0 and the work order modification quality value is equal to 0, the work order quality level is level 5.

3. The method for allocating text annotation tasks based on artificial intelligence according to claim 2 is characterized in that: The step of constructing a labeling personnel determination model, determining the work order type of the work order to be assigned through the labeling personnel determination model, obtaining historical work orders corresponding to the work order type, obtaining data labeling personnel corresponding to the historical work orders, and selecting a person to be assigned to the work order based on the data labeling personnel includes: The labeling personnel determine the model, determine the work order type of the work order to be assigned, obtain the historical work order corresponding to the work order type, obtain the data labeling personnel corresponding to the historical work order, and determine the data labeling personnel with a quality level of 3 or above as the personnel to be assigned to the work order.

4. The method for allocating text annotation tasks based on artificial intelligence according to claim 1 is characterized in that: The step of constructing a labeling personnel determination model, determining the work order type of the work order to be assigned through the labeling personnel determination model, obtaining historical work orders corresponding to the work order type, obtaining data labeling personnel corresponding to the historical work orders, and selecting a person to be assigned to the work order based on the data labeling personnel, further includes: Analyze the annotators through test questions and obtain their scores based on different dimensions; Obtain historical data of all labelers on the platform, and predict the types of labeling work orders that the labelers are good at based on the historical data; The best work order type for the first assignment of the labeler is obtained based on the labeler's scores based on different dimensions and the labeling work order types that the labeler is good at.

5. The method for allocating text annotation tasks based on artificial intelligence according to claim 4 is characterized in that: The step of constructing a labeling personnel determination model, determining the work order type of the work order to be assigned through the labeling personnel determination model, obtaining historical work orders corresponding to the work order type, obtaining data labeling personnel corresponding to the historical work orders, and selecting a person to be assigned to the work order based on the data labeling personnel, further includes: During secondary allocation, different work order types with the same complexity level are allocated.

6. The method for allocating text annotation tasks based on artificial intelligence according to claim 1, characterized in that: The complexity of obtaining a certain number of historical work orders includes: Based on the field information to be extracted from the work order requirements, at least one triple is constructed, the relationship between the triples is analyzed, and based on the relationship between the triples and the number of triples, the entity relationship complexity of the work order to be assigned is obtained; The text content complexity is calculated by formula (4); Text content complexity = (number of sentences / number of characters)*100 Formula (4); The amount of text content is calculated by formula (5); Text content = number of text characters / 1000; The complexity of the work order to be assigned is obtained through the entity relationship complexity, text content complexity and text content volume.

7. The method for allocating text annotation tasks based on artificial intelligence according to claim 1, characterized in that: The step of storing the personnel to be assigned to the work order in a pool of personnel to be screened, determining whether the number of rows of the work order to be assigned is greater than a preset threshold value N1, and if so, selecting personnel to be assigned to the work order with a fast marking speed from the pool of personnel to be screened; and determining whether the number of rows of work order data is less than a preset threshold value N2, and if so, selecting personnel to be assigned to the work order with a high marking quality level from the pool of personnel to be screened, includes: When the number of work order rows is greater than the preset threshold value N1, the work order data of the work order type corresponding to the personnel to be assigned for each work order in the pool of personnel to be screened and assigned within a certain time period is obtained; The average rate V of each work order type being processed by the personnel to be assigned to each work order is calculated by formula (6); The average rate V of each work order type being processed by the personnel to be assigned to the work order = weighted average ((number of work order data rows * complexity of entity relationships of the work order * complexity of work order text content * amount of work order content) / actual time required to mark the work order) formula (6); Select the person to be assigned the work order with the largest average rate V. If there are multiple people, select the person to be assigned the work order with the earliest completion time node; When the number of work order rows is less than the preset threshold value N2, the work order data of the work order type corresponding to each work order to be assigned personnel in the pool of personnel to be screened and assigned within a certain period is obtained; The average quality R of each work order type handled by the personnel to be assigned to each work order is calculated by formula (7); Average quality R = weighted average (work order quality level) formula (7); Select the person to be assigned the work order with the largest average quality R. If there are multiple people, select the person to be assigned the work order with the earliest completion time node.

8. A text annotation task allocation system based on artificial intelligence, characterized in that: include: Text annotation quality analysis module, used to obtain the quality level of a certain number of historical work orders; A work order complexity calculation module is used to obtain the complexity of a certain number of historical work orders, obtain the work orders to be assigned, calculate the work orders to be assigned, and obtain the complexity of the work orders to be assigned; A text annotation time prediction module is used to train the work order annotation time prediction model based on the complexity of historical work orders and the time range required for the completion of historical work orders as inputs to obtain an optimal work order annotation time prediction model; The labeling personnel feature testing module is used to build a labeling personnel determination model, determine the work order type of the work order to be assigned through the labeling personnel determination model, obtain the historical work order corresponding to the work order type, obtain the data labeling personnel corresponding to the historical work order, and select the personnel to be assigned to the work order based on the data labeling personnel; A text annotation task assignment module is used to input the complexity of the work order to be assigned into the optimal work order annotation time prediction model to obtain the expected time range [X, X1] for completing the work order to be assigned; The completion time node of the work order to be assigned to the personnel is calculated by formula (1); Z1=Z0+Y1+Y2+X1 formula (1); In the formula, Z1 is the completion time node of the person to be assigned the work order, Z0 is the current time node, Y1 is the intermittent rest time between work orders within a certain time period for the person to be assigned the work order, Y2 is the sum of the upper limit time of the time range required for the work order corresponding to the current work order to be processed by the person to be assigned the work order, and X1 is the upper limit time required for the work order to be assigned to complete the work order; Determine whether Z1 exceeds the delivery time node Z required by the work order. If so, re-acquire the personnel to be assigned to the work order until the completion time node of the personnel to be assigned to the work order does not exceed the delivery time node required by the work order; If not, store the personnel to be assigned to the work order into the pool of personnel to be screened and assigned, and determine whether the number of work order rows to be assigned is greater than the preset threshold value N1. If so, select the personnel to be assigned to the work order with a fast marking speed from the pool of personnel to be screened and assigned; determine whether the number of work order data rows is less than the preset threshold value N2. If so, select the personnel to be assigned to the work order with a high marking quality level from the pool of personnel to be screened and assigned.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A non-transitory computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images