Labeling task allocation method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides an annotation task allocation method and device, equipment, a storage medium and a program product. The method comprises the following steps: acquiring a plurality of annotation tasks to be distributed; determining task portraits of the plurality of to-be-distributed annotation tasks according to the plurality of to-be-distributed annotation tasks; obtaining capability portraits of a plurality of annotation personnel and capability portraits of a plurality of data annotation models, wherein the data annotation models are used for data annotation; according to the task portrait, the ability portraits of the plurality of annotation personnel and the ability portraits of the plurality of data annotation models, constructing an optimization model used for determining an allocation relationship among the to-be-allocated annotation task, the annotation personnel and the data annotation models; and solving the optimization model by using a solving algorithm to generate a target task allocation scheme.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a method, apparatus, storage medium, and program product for assigning labeling tasks. Background Technology

[0002] Currently, with the rapid development of artificial intelligence (AI) and big data technologies, data labeling, as a key step in AI model training, is becoming increasingly important.

[0003] In crowdsourced annotation and data processing platforms, as task types and processing methods become increasingly complex, achieving efficient and intelligent allocation of annotation tasks is key to improving platform operational efficiency and quality.

[0004] Therefore, there is an urgent need to provide an efficient and intelligent task allocation solution. Summary of the Invention

[0005] This specification provides a method, apparatus, device, storage medium, and program product for assigning tasks, which enables intelligent task assignment and improves the rationality of task assignment.

[0006] The first aspect of this specification provides a method for assigning annotation tasks, including: Get multiple labeling tasks to be assigned; Based on the multiple unassigned annotation tasks, determine the task profiles of the multiple unassigned annotation tasks; Obtain capability profiles of multiple annotators and multiple data annotation models, wherein the data annotation models are used for data annotation; Based on the task profile, the capability profiles of the multiple annotators, and the capability profiles of the multiple data annotation models, an optimization model is constructed to determine the allocation relationship between the annotation tasks to be assigned, the annotators, and the data annotation models. The optimization model is solved using a solution algorithm to generate a target task allocation scheme.

[0007] A second aspect of this specification provides a task assignment device, comprising: The first acquisition module is used to acquire multiple annotation tasks to be assigned. The determination module is used to determine the task profile of the multiple assignment tasks based on the multiple assignment tasks; The second acquisition module is used to acquire the capability profiles of multiple annotators and multiple data annotation models, wherein the data annotation models are used for data annotation. The construction module is used to construct an optimization model based on the task profile, the capability profiles of the multiple annotators, and the capability profiles of the multiple data annotation models to determine the allocation relationship between the annotation tasks to be assigned, the annotators, and the data annotation models. The solution module is used to solve the optimization model using a solution algorithm to generate a target task allocation scheme.

[0008] A third aspect of this specification provides an electronic device, comprising: a memory and a processor, wherein, The memory is used to store programs; The processor, coupled to the memory, is configured to execute the program stored in the memory to implement the method described in any of the preceding embodiments.

[0009] A fourth aspect of this specification provides a computer-readable storage medium storing a computer program that, when executed by a computer, enables the implementation of any of the methods described above.

[0010] A fifth aspect of this specification provides a computer program product, including a computer program that, when executed by a processor, implements the method described in any of the preceding descriptions.

[0011] The technical solution provided in this application establishes a task allocation optimization model based on multi-dimensional profiles by characterizing the capabilities of annotation personnel and data annotation models, combined with task profiles. This optimization model can dynamically weigh the capabilities of annotation personnel and data annotation models, rationally allocate tasks, avoid wasting human resources or leaving data annotation models idle, and significantly improve overall execution performance. This optimization model receives personnel profiles, model profiles, and task profiles, transforming the task allocation process into a computable mathematical optimization problem, thereby achieving efficient, rational, and interpretable resource allocation, suitable for high-reliability application scenarios. Attached Figure Description

[0012] The accompanying drawings, which are provided to further illustrate this specification, form part of this specification.

[0013] Figure 1 A flowchart illustrating an exemplary embodiment of the annotation task allocation method provided in this specification; Figure 2 A flowchart illustrating a labeling task allocation method provided as yet another exemplary embodiment of this specification; Figure 3 This is a schematic diagram of the structure of an electronic device provided as another exemplary embodiment of this specification. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments, and not all of the embodiments. Based on these embodiments, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0015] It should be noted that, in the cases involving user information in the embodiments of this specification, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse. In addition, the various models involved in this specification (including but not limited to language models or large models) comply with relevant laws and standards.

[0016] Before introducing the specific details of each embodiment in this specification, a brief explanation of the technical terms mentioned herein will be provided.

[0017] A profile is a digital representation of an entity (such as a person, data-annotated model, or task) based on structured modeling of its capabilities, attributes, behavioral characteristics, or functional features using multi-source data. Specifically, a profile consists of multiple quantifiable dimensions, each corresponding to a measurable indicator, and mapped to a unified numerical value through a standardized scoring system. This supports cross-entity comparison, matching, and decision analysis. The aforementioned profile is not simply a summary of information, but rather structured input data generated through data collection, feature extraction, weight calculation, and scoring mapping. It serves as the core technological foundation for intelligent task allocation, resource scheduling, and human-machine collaborative decision-making.

[0018] A language model (LM) is a deep learning model trained on data, typically possessing broad task adaptability and reasoning capabilities, such as the Transformer architecture. These models can understand complex contextual information and generate accurate predictions and suggestions in scenarios such as data analysis and decision support. The embodiments in this specification do not limit the number of model parameters supported by the language model, aiming to meet application requirements. If the model has relatively more parameters, the language model will be relatively larger and its performance will be relatively better; however, it will consume more time and resources during inference and training. If the model has relatively fewer parameters, the language model will be relatively smaller and, while meeting performance requirements, more lightweight, consuming less time and resources during inference and training. When the number of model parameters of a language model is greater than or equal to a preset threshold, the language model can be called a Large Language Model (LLM).

[0019] An optimization model is a decision support system built using mathematical programming methods. It is used to find the extreme values ​​of an objective function within a given feasible solution space. An optimization model includes decision variables, an objective function, and constraints. Decision variables are the unknowns (controllable factors) that need to be solved in the optimization problem, and their values ​​directly affect the objective function. The objective function, expressed in functional form, represents the goal pursued by the decision-maker. It is a mathematical expression used in the optimization model to quantitatively evaluate the merits of different decision options, and its values ​​depend on the decision variables and pre-defined model parameters. Constraints are the limiting conditions that the decision variables must satisfy, or mathematical expressions that restrict the values ​​of the decision variables or the relationships between variables.

[0020] Genetic Algorithms (GAs) are optimization algorithms that simulate the principles of natural selection and genetics, and are widely used to solve complex optimization problems. These algorithms use mathematical methods and computer simulations to transform the problem-solving process into processes similar to the selection, crossover, and mutation of chromosomes and genes in biological evolution. NSGA-II (Non-dominated Sorting Genetic Algorithm II) is a commonly used multi-objective optimization algorithm widely applied to solving multi-objective optimization problems. The adaptive NSGA-II algorithm is an improvement on the standard NSGA-II algorithm. In the adaptive NSGA-II algorithm, crossover and mutation probabilities are dynamically adjusted, ensuring population diversity and exploration capabilities while propelling the algorithm towards a globally optimal solution.

[0021] The Pareto front refers to the set of solutions in a multi-objective optimization scenario where it is impossible to further optimize one objective without harming the other objectives. This set of solutions can also be called the non-dominated solution set.

[0022] Figure 1 This is a flowchart illustrating a labeling task allocation method provided in one embodiment of this specification. The execution entity of this method can be a terminal device or a server device. The terminal device may include, but is not limited to, smartphones, smart wearable devices, tablets, laptops, desktop computers, etc. The server device may include, but is not limited to, servers, service clusters, virtual servers, or the cloud, etc., and this embodiment does not specifically limit these. Figure 1 As shown, the method may include the following steps: 100. Obtain multiple labeling tasks to be assigned.

[0023] 102. Based on the multiple unassigned labeling tasks, determine the task profiles of the multiple unassigned labeling tasks.

[0024] 104. Obtain capability profiles of multiple annotators and multiple data annotation models.

[0025] 106. Based on the task profile, the capability profiles of the multiple annotators, and the capability profiles of the multiple data annotation models, construct an optimization model for determining the allocation relationship between the annotation tasks to be assigned, the annotators, and the data annotation models.

[0026] 108. Solve the optimization model using a solution algorithm to generate a target task allocation scheme.

[0027] Among the 100 mentioned above, the objects to be labeled in the assigned labeling tasks may include one or more data modalities such as text, images, and videos.

[0028] In practical applications, a labeling task to be assigned may contain one or more objects to be labeled, and the modal types of the objects to be labeled may be one or more.

[0029] Optionally, preprocessing can be performed on the text data to be assigned to the annotation task. This not only helps with subsequent image generation but also with subsequent annotation. For example, semantic segmentation and entity recognition can be performed on the text data in the annotation task, and the segmentation and entity recognition results can be added to the annotation task for reference during subsequent annotation. Standardized resizing and noise filtering can be implemented on the image data in the annotation task. The video data in the annotation task can be decomposed into a series of keyframes.

[0030] Optionally, domain-adaptive algorithms can be used to improve data annotability. For example, medical images can be automatically enhanced to increase the contrast of lesion areas, and legal texts can be analyzed for clause relevance. This not only reduces the difficulty of subsequent annotation but also improves its accuracy.

[0031] In step 102 above, each unassigned labeling task can be analyzed to determine the task profile of each unassigned labeling task.

[0032] Optionally, the task to be labeled is analyzed according to multiple preset dimensions to obtain the indicators of the task to be labeled in these multiple dimensions; based on the indicators of the task to be labeled in these multiple dimensions, a task profile of the task to be labeled is generated.

[0033] The aforementioned preset dimensions may include: domain type, task type, task complexity, required skill type, and task requirements. Task requirements can be further subdivided into quality requirements, time constraints, etc. Task complexity can be further subdivided into multiple sub-dimensions, such as: data volume, data modality, number of label levels, annotation process complexity, etc.

[0034] In some embodiments, the task profile may include information such as domain type, task type, task complexity, required skill types, quality requirements, and time constraints. The task profile can be output as a file to guide subsequent task allocation.

[0035] In the above 104, the data annotation model is used for data annotation. The data annotation model can be an AI-based language model. For example, the data annotation model can be a large language model.

[0036] In practical applications, the annotation abilities of different annotators vary, therefore, the capability profiles of different annotators differ. Similarly, the annotation capabilities of different data annotation models vary, therefore, the capability profiles of different data annotation models differ.

[0037] In some embodiments, the competency profile of annotators may include, but is not limited to, dimensions such as domain competence, annotation quality, annotation efficiency, cost, learning ability, and human-computer collaboration ability. For example, a competency profile of annotators can be constructed by collecting data from various sources, such as work experience, knowledge background, training information, professional certifications, historical annotation quality, historical annotation efficiency, and historical human-computer collaboration. Human-computer collaboration ability refers to annotators' ability to use data annotation models as auxiliary tools. Different annotators typically have varying abilities to use different data annotation models as auxiliary tools. For example, some annotators are proficient in using data annotation model A, while others are proficient in using data annotation model B. Domain competence can be presented as a matrix to represent annotators' competency values ​​in different domains.

[0038] A capability profile for a data annotation model can include, but is not limited to, domain capability, annotation quality, annotation efficiency, service stability, resource consumption cost, knowledge timeliness, and confidence decay. This profile can quantify the model's performance and resource consumption characteristics in different scenarios. Annotation efficiency is generally related to the model's parameter size; the larger the parameter size, the lower the annotation efficiency, and vice versa. Knowledge timeliness can be understood as the time since the model's most recent fine-tuning or iteration, and confidence decay refers to the decay information of the model's output confidence. Output confidence can be obtained by verifying the model's output. For example, this can be achieved by collecting various data, such as: the model's training or fine-tuning results and training or fine-tuning time points, the domain of the model's training data, the model's historical annotation quality, annotation time, resource consumption information, and verification information of the model's output annotation results. Domain capability can be a matrix representing the data annotation model's capability values ​​in different domains.

[0039] Alternatively, the capability profiles of the aforementioned labelers and data labeling models can be pre-calculated and directly obtained during subsequent task allocation.

[0040] Optionally, the capability profiles of the aforementioned annotators and data annotation models can be updated at a preset frequency (e.g., once a week, once a day, etc.) or triggered by specific events to ensure the timeliness of the profiles. Taking model profile updates as an example, specific events may include events such as the completion of model fine-tuning or iteration, or the detection that the confidence decay of the model (e.g., a decay of 40%) is greater than a preset threshold.

[0041] In the above 106, the allocation relationship between the annotation task to be assigned, the annotation personnel and the data annotation model may include: the allocation relationship between the annotation task to be assigned and the annotation personnel, the allocation relationship between the annotation task to be assigned and the data annotation model, and the allocation relationship between the personnel to be annotated and the data annotation model.

[0042] The solution of the optimization model (i.e. the solution of the objective function) is used to represent the task allocation scheme.

[0043] Based on the multiple unassigned annotation tasks, the multiple annotation personnel, and the multiple data annotation models, an objective function is constructed, and based on the objective function, an optimization model is constructed with the objective function as the goal.

[0044] Optionally, constraints can be determined based on allocation rules and / or task profiles, such as: a task can be assigned to at most one labeler and / or one data labeling model, total cost constraints, etc.

[0045] An optimization model is constructed based on the objective function and constraints.

[0046] In some embodiments, the optimization model described above can be used to optimize a performance metric, such as annotation cost, annotation time, and annotation quality. The objective function of this optimization model is a single function.

[0047] In some embodiments, the optimization model described above is used to optimize multiple performance metrics, including at least two of annotation cost, annotation time, and annotation quality. The optimization model has multiple objective functions, with different objective functions used to optimize different performance metrics.

[0048] In step 108 above, the optimization model is solved using a solution algorithm to obtain the solution results; based on the solution results, the target task allocation scheme is determined.

[0049] When the objective function of the above optimization model is used for single-objective optimization, the solution result generally contains a solution, and the task allocation scheme represented by the solution can be directly used as the target task allocation scheme.

[0050] When the objective function of the above optimization model is used for multi-objective optimization, the solution result contains multiple solutions. In one optional implementation, any one of these multiple solutions can be used as the target task allocation scheme. In another optional implementation, the target task allocation scheme can be selected from the multiple task allocation schemes represented by the multiple solutions according to a preset decision strategy. The preset decision strategy can be set according to actual needs, and this specification does not specifically limit it in the embodiments.

[0051] The technical solution provided in this application establishes a task allocation optimization model based on multi-dimensional profiles by characterizing the capabilities of annotation personnel and data annotation models, combined with task profiles. This optimization model can dynamically weigh the capabilities of annotation personnel and data annotation models, rationally allocate tasks, avoid wasting human resources or leaving data annotation models idle, and significantly improve overall execution performance. This optimization model receives personnel profiles, model profiles, and task profiles, transforming the task allocation process into a computable mathematical optimization problem, thereby achieving efficient, rational, and interpretable resource allocation, suitable for high-reliability application scenarios.

[0052] In the above embodiments, the capability profile can be understood as a long-term profile. To enhance the timeliness of the profiles of annotators and models, their capability profiles can be supplemented based on the current status information of the annotators, such as annotation quality, workload, fatigue status, quality fluctuation information, and efficiency fluctuation information. Similarly, their capability profiles can be supplemented based on the current status of the data annotation model, such as annotation quality, service stability, workload, operational status, and confidence decay.

[0053] In some embodiments, the step 106 above, "constructing an optimization model based on the task profile, the capability profiles of the multiple annotators, and the capability profiles of the multiple data annotation models," can be implemented using the following steps: 1060. Determine decision variables based on the multiple unassigned annotation tasks, the multiple annotation personnel, and the multiple data annotation models.

[0054] 1062. Based on the task profile, the capability profiles of the multiple annotators, and the capability profiles of the multiple data annotation models, generate performance evaluation parameters.

[0055] 1064. Construct an objective function based on the performance evaluation parameters and the decision variables.

[0056] 1066. Based on the objective function, construct an optimization model with the objective function as the goal.

[0057] In the above 1060, the decision variable is used to indicate whether the allocation relationship between the assignment task, the labeler, and the data labeling model is valid. The decision variable can take a first value (e.g., 1) and a second value (e.g., 0). When the decision variable takes the first value, it indicates that the allocation relationship corresponding to the decision variable is valid; when the decision variable takes the second value, it indicates that the allocation relationship corresponding to the decision variable is invalid.

[0058] For example, let's define the decision variable as X[i, j, k], which represents whether to assign the i-th annotation task to the j-th annotator and use the k-th data annotation model. Assigning the i-th annotation task to the j-th annotator and using the k-th data annotation model can be understood as the allocation relationship between the i-th annotation task, the j-th annotator, and the k-th data annotation model.

[0059] Assume the number of annotation tasks to be assigned is I, the number of annotators is J, and the number of data annotation models is K, where I, J, and K are all integers greater than 1.

[0060] In some embodiments, the value of i ranges from 1 to I, where i is an integer; the value of j ranges from 1 to J, where j is an integer; and the value of k ranges from 1 to K, where k is an integer. In this embodiment, the task has only one processing mode: human-machine collaborative processing mode. That is, each task is performed by annotators with the assistance of a data-assisted model.

[0061] In practical applications, some tasks are quite complex, and the model's understanding or knowledge of these tasks is limited. In such cases, if processed using a human-machine collaborative model, the annotator may be misled by the model's incorrect assistance, leading to erroneous annotations or negatively impacting the annotator's efficiency. In other words, these tasks are best performed by the annotator alone. Other tasks are simpler, and some capable models can handle them without the need for annotators, thus fully utilizing the model and reducing labor costs. Therefore, in some embodiments, the value of i ranges from 1 to I, where i is an integer; the value of j ranges from 0 to J, where j is an integer; and the value of k ranges from 0 to K, where k is an integer. Here, j=0 represents no annotator, and k=0 represents a model without data annotation. That is, a non-entity of "no annotator" is added to the set of annotators, and a non-entity of "no data annotation model" is added to the set of data annotation models.

[0062] In this embodiment, the task has three processing modes: a manual mode where annotators perform the annotation task without the participation of a data annotation model; an automatic mode where the data annotation model performs the annotation task without the participation of annotators; and a human-machine collaborative mode where annotators and the data annotation model jointly participate in the annotation task. X[i, j, 0] represents the manual mode, X[i, 0, k] represents the automatic mode, and X[i, j, k] (where j > 0 and k > 0) represents the human-machine collaborative mode.

[0063] In the above 1062, the performance evaluation parameters are related to the desired optimization objective, that is, to the performance metric to be optimized. If the desired performance metric includes annotation quality, then the performance evaluation parameters include quality evaluation parameters. If the desired performance metric includes annotation cost, then the performance evaluation parameters include cost evaluation parameters. If the desired performance metric includes annotation time, then the performance evaluation parameters include time evaluation parameters.

[0064] In some embodiments, performance evaluation parameters corresponding to the above allocation relationship can be generated based on the task profile, the capability profiles of multiple annotators, and the capability profiles of multiple data annotation models.

[0065] For example, the cost assessment parameter is defined as c_ijk, which represents the estimated cost required for the i-th task to be completed by the j-th annotator with the assistance of the k-th data annotation model.

[0066] For example, the quality assessment parameter is defined as q_ijk, which represents the estimated quality required for the i-th task to be completed by the j-th annotator with the assistance of the k-th data annotation model.

[0067] For example, the time evaluation parameter is defined as s_ijk, which represents the estimated time required for the i-th task to be completed by the j-th annotator with the assistance of the k-th data annotation model.

[0068] In the above 1064, for example, if the performance metric to be optimized includes annotation cost, then the objective function for optimizing annotation cost is constructed as formula (1): (1) For example, if the performance metric to be optimized includes annotation quality, then the objective function for optimizing annotation quality is constructed as formula (2): (2) For example, if the performance metric to be optimized includes annotation time, then the objective function for optimizing annotation time is constructed as formula (3): (3) This objective function optimizes the working time of the person who finishes the annotation last in the set of personnel.

[0069] In step 1066 above, in addition to the objective function, an optimization model can also be constructed based on constraints.

[0070] The constraints may include: each task must be completed once (exactly once), personnel load constraints (e.g., maximum working hours for each person), each task can be assigned to at most one person and / or one model, etc.

[0071] This embodiment provides a specific implementation method for constructing an optimization model based on multi-dimensional profiles. Furthermore, by adding two non-entities—unlabeled personnel and unlabeled data models—it enables multiple task processing modes: manual model, automatic mode, and manual collaboration mode, thereby fully optimizing resource utilization.

[0072] In some embodiments, the algorithm for solving the optimization model may include, but is not limited to, gradient descent, genetic algorithm, and ant colony algorithm. The optimization model provided in the embodiments of this specification will be solved using a genetic algorithm below. Step 108 above, "solving the optimization model using a solution algorithm to generate a target task allocation scheme," can be implemented using the following steps: 1080a. Encode the solution of the optimization model to obtain the encoded solution.

[0073] 1082a. Based on the encoded solution, the genetic algorithm is used for iterative search to generate a target task allocation scheme.

[0074] In the aforementioned 1080a, the solution of the optimization model is used to represent the potential task allocation scheme of the optimization model. The potential task allocation scheme of the optimization model can be encoded to obtain the encoded solution, which can be called a chromosome.

[0075] The encoded solution includes gene units constructed for each labeling task to be assigned. The gene units include: mode selection gene, personnel selection gene, and model selection gene. The mode selection gene is used to indicate the processing mode of the labeling task to be assigned. The personnel selection gene is used to indicate the assigned labeling personnel when the processing mode indicates that personnel participation is required. The model selection gene is used to indicate the assigned data labeling model when the processing mode indicates that data labeling model participation is required.

[0076] The processing mode includes at least two of the following three modes: a manual mode in which annotators perform annotation tasks without the participation of a data annotation model; an automatic mode in which the data annotation model performs annotation tasks without the participation of annotators; and a human-machine collaborative mode in which annotators and data annotation models jointly participate in annotation tasks.

[0077] For example, the above encoded solution can be represented in binary, where the pattern gene can be represented by two binary bits, the personnel selection gene can be represented by J+1 binary bits, and the model selection gene can be represented by K+1 binary bits.

[0078] In some embodiments, an initial population can be generated, for example, by randomly generating N chromosomes that satisfy the constraints to construct the initial parent population.

[0079] In the above 1082a, based on the initial parent population, the genetic algorithm is used for iterative search, so that the population continues to evolve to obtain the target task allocation scheme.

[0080] Starting with an initial parent population, N offspring individuals are generated through selection, crossover, and mutation to form the offspring population. Based on an elite preservation strategy, the parent and offspring populations are merged, and the individuals in the merged population are sorted from highest to lowest fitness. The top N individuals are selected to form a new parent population, serving as the starting point for the next iteration. This process is repeated until a predetermined number of iterations is reached or population evolution stagnates (e.g., multiple iterations show no significant improvement in the optimal solution). When the algorithm stops, the individuals in the current population are sorted from highest to lowest fitness, and the individual with the highest fitness in the current population is selected as the optimal solution. The task allocation scheme represented by the optimal solution is the target task allocation scheme.

[0081] In some embodiments, the genetic algorithm may include: the NSGA-II algorithm or the adaptive NSGA-II algorithm.

[0082] This embodiment utilizes a genetic algorithm to solve the optimization model. By simulating natural selection and genetic mechanisms, it can efficiently and in parallel search large-scale and complex solution spaces, effectively approximating the globally optimal solution. The algorithm possesses both powerful global exploration and local fine-grained search capabilities, effectively handling various constraints and avoiding getting trapped in local optima. For optimization problems such as task allocation, the genetic algorithm exhibits excellent robustness and scalability, obtaining high-quality and feasible task allocation schemes without relying on problem-specific knowledge. In some embodiments, the optimization model is used to optimize multiple performance indicators. Step 108 above, "solving the optimization model using a solution algorithm to generate a target task allocation scheme," can be implemented using the following steps: 1080b. Solve the optimization model using a solution algorithm to obtain the Pareto solution set.

[0083] The Pareto solution set includes multiple solutions, which represent various alternative task allocation schemes.

[0084] 1082b. Determine the target task allocation scheme from the multiple alternative task allocation schemes.

[0085] In the aforementioned 1080b algorithm, taking a genetic algorithm as an example, based on an initial parent population, N offspring individuals are generated through selection, crossover, and mutation to form an offspring population. Based on an elite preservation strategy, the parent and offspring populations are merged. Using non-dominated sorting and crowding sorting methods, the better N individuals are selected from the merged population (2N individuals) to form a new parent population, which serves as the starting point for the next iteration. The preceding steps are repeated until a preset number of iterations is reached or the population evolution stagnates, at which point the algorithm stops. After the algorithm completes, the first non-dominated layer (F1) in the final parent population is the Pareto solution set found by the algorithm.

[0086] In the above 1082b, the target task allocation scheme can be randomly determined from the multiple alternative task allocation schemes; or, the target task allocation scheme can be screened from the multiple alternative task allocation schemes according to a preset decision strategy. This decision strategy can be set according to actual needs. For example, if the decision-maker prefers the annotation quality, then the task allocation scheme with better annotation quality is selected. Or, if the decision-maker prefers the annotation time, then the task allocation scheme with shorter time consumption is selected.

[0087] In an optional implementation, the step of "determining the target task allocation scheme from the multiple alternative task allocation schemes" in 1082b above can be achieved by the following steps: S11. Based on the task labeling quality of the processed tasks, the resource consumption information of the processed tasks, and / or the time limit of the multiple labeling tasks to be assigned, determine the weights corresponding to the multiple performance indicators.

[0088] S12. Determine the performance score of the multiple alternative task allocation schemes based on the weights corresponding to the multiple performance indicators and the index values ​​of the multiple alternative task allocation schemes under each performance indicator.

[0089] S13. Based on the performance score, select the target task allocation scheme from the multiple alternative task allocation schemes.

[0090] In S11 above, the processed task and one or more of the tasks among the multiple unassigned annotation tasks belong to the same annotation project. Generally speaking, users will set annotation quality requirements and resource budget limits for annotation projects.

[0091] When the annotation quality of a processed task deviates from the target requirements, the weight of the corresponding quality indicator can be increased.

[0092] When resource consumption approaches the resource budget limit for the entire project, the weight of the corresponding cost indicators can be increased.

[0093] When there are tasks with close deadlines among multiple assignment tasks, the weight of the time metric can be increased.

[0094] In S12 above, the index values ​​of the alternative task allocation scheme under each performance index are weighted and summed according to the weights corresponding to the various performance indicators to obtain the performance score of the alternative task allocation scheme.

[0095] In S13 above, the alternative task allocation scheme with the higher performance score can be used as the target task allocation scheme.

[0096] This embodiment balances quality requirements and resource consumption in real time, and dynamically adjusts the strategy of selecting target allocation schemes from the Pareto solution set according to the system status.

[0097] After obtaining the target task allocation plan, tasks can be allocated according to the plan. For example, task A in manual mode can be assigned to annotator a, task B in automatic mode can be assigned to data annotation model b, and task C in human-machine collaboration mode can be assigned to annotator c and data annotation model d. In human-machine collaboration mode, specific collaboration methods can include guided and parallel modes. Guided mode refers to the large model generating initial annotation results, which are then refined by human annotators, with the system providing real-time auxiliary suggestions (such as highlighting discrepancies and compliance checks). Parallel mode refers to humans and machines independently completing the same task, with the system automatically merging the results and highlighting conflicting areas for expert arbitration. Conflict detection algorithms can be used for conflict detection, such as using feature similarity comparison based on Siamese networks to set dynamic thresholds to determine discrepancies in annotation results.

[0098] To enhance the annotation experience in human-machine collaborative modes, an interactive annotation interface supporting collaboration between humans and AI can be provided, enabling the integration of intelligent assistance and decision-making. This interactive annotation interface highlights areas of inconsistency between human and AI annotations, assisting humans in quickly locating points of contention and displaying annotation suggestions. The data annotation model can dynamically generate annotation suggestions based on human input (such as boundary fine-tuning prompts), and displays a collaboration mode switching control to support hybrid modes such as "human-led - AI-assisted," "AI-led - human verification," and "parallel independent annotation."

[0099] In this embodiment, by designing a labeling interface that conforms to cognitive ergonomics, operational fatigue and cognitive load in human-machine collaboration are reduced.

[0100] In practical applications, after the task allocation is carried out according to the above target task allocation scheme, it is also possible to detect whether there are any abnormalities in the labeling personnel or data labeling models. If there are any abnormalities, the task allocation is fully implemented or the task is downgraded to achieve a flexible fault tolerance mechanism.

[0101] In some embodiments, the above method may further include: 110. Verify the data annotation model and the annotation results output by the annotators to obtain verification information.

[0102] Optionally, a hierarchical quality verification method is employed to verify the annotation results output by the data annotation model, obtaining verification information. The hierarchical quality verification method involves three layers: a basic verification layer, a cross-validation layer, and an arbitration decision layer. Specifically, the basic verification layer performs rule-based checks such as format compliance checks and value range rationality verification. The cross-validation layer uses a voting mechanism (consistency must be ≥2 / 3) to determine the final verification result after multiple large models have independently verified the data. The arbitration decision layer involves domain experts conducting a final review of annotation results that cannot be accurately verified by the large models or high-risk annotations (such as legal clause boundary determination). The decision basis of the domain experts during the final review is stored in a rule knowledge base.

[0103] Typical dispute cases accumulated during the verification process can be transformed into a rule base, and the verification algorithm can be continuously optimized to improve the large model used for verification.

[0104] In some embodiments, personnel profiles and model profiles can be updated. For example, the skill development trajectory of labeled personnel (such as their mastery of new domain knowledge) can be tracked, dynamically expanding the range of tasks they can undertake. Changes in the service performance of the data labeling model (such as capability drift after version upgrades) can be detected, and its domain capability assessment data can be updated in real time.

[0105] In some embodiments, the above method may further include the following steps: 112. Update the capability profile of the data annotation model based on the annotation data of the data annotation model.

[0106] The capability profile of the data annotation model can be updated based on the annotation data collected within a recent period (e.g., within a week, a day, or an hour). The annotation data includes annotation quality and annotation efficiency, which can be updated using a sliding window algorithm and / or a decaying silver model.

[0107] The technical solution provided in this embodiment updates the model profile based on the model's annotation feedback, thereby improving the timeliness of the profile.

[0108] In some embodiments, the above method may further include the following steps: 114. Update the capability profile of the annotators based on their annotation data.

[0109] The capability profile of the data annotation model can be updated based on the annotation data collected by the annotators within a recent period (e.g., within a week, a day, or an hour). The annotation data includes annotation quality and annotation efficiency, which can be updated using a sliding window algorithm and / or a decaying silver model.

[0110] The technical solution provided in this embodiment updates personnel profiles based on personnel annotation feedback, thereby improving the timeliness of the profiles.

[0111] In some embodiments, the above method may further include: 116. Construct training data based on the historical annotation results of the annotators and / or the historical annotation results of the data annotation model.

[0112] The training data includes positive samples and negative samples.

[0113] 118. Based on the training data, the data labeling model is fine-tuned using a contrastive learning algorithm.

[0114] In step 116 above, negative samples can be constructed based on historical annotation results whose annotation quality does not meet preset quality requirements. For example, the historical annotation result and its corresponding first annotation object (e.g., text, image, video) can be used as negative samples. Positive samples can be constructed based on historical annotation results whose annotation quality meets preset quality requirements. For example, the historical annotation result and its corresponding second annotation object can be used as positive samples.

[0115] In step 118 above, the first and second labeled objects can be input into the data labeling model. The data labeling model outputs the labeling results of the first and second labeled objects. A first distance is determined between the first labeled object and its historical labeling results based on the similarity between the first labeled object's labeling results and its historical labeling results. A second distance is determined between the second labeled object and its historical labeling results based on the similarity between the second labeled object's labeling results and its historical labeling results. Based on the first and second distances, a contrastive loss function is used to calculate the loss value, and the parameters of the data labeling model are fine-tuned based on the loss value. The optimization purpose of this contrastive loss function is to make the model's output closer to positive samples and further away from negative samples.

[0116] LoRA (Low-Rank Adaptation) fine-tuning technique can be used to fine-tune the model parameters based on the loss value.

[0117] In this embodiment, the contrastive learning algorithm is used to fine-tune the model, which can not only reduce the cost of building training data, but also improve the accuracy of model annotation and achieve targeted enhancement of the capabilities of large models.

[0118] In practical applications, after the model is fine-tuned, in order to ensure service continuity, the new fine-tuned model is gradually replaced by the old version after being verified on a small scale, thereby realizing the version canary release.

[0119] In some embodiments, after the data annotation model has been fine-tuned, the capability profile of the data annotation model is updated based on the fine-tuning results. The fine-tuning results may include information on improved annotation quality and improved annotation efficiency.

[0120] In this embodiment, the domain capability assessment data can be updated in a timely manner after the model version is upgraded.

[0121] In some embodiments, the above method may further include: 120. Obtain the quality verification information of the annotation results of the data annotation model.

[0122] 122. Based on the quality verification information, determine the degree of confidence decay of the data annotation model.

[0123] 124. Based on the degree of confidence decay, determine whether the data labeling model needs to be fine-tuned.

[0124] In the above 120, the above verification method can be used to verify the quality of the annotation results output by the data annotation model, and obtain quality verification information. The quality verification information may include confidence level.

[0125] In step 122 above, the first confidence level in the quality verification information can be compared with the second confidence level obtained after the last model fine-tuning of the data annotation model to obtain the degree of confidence decay of the data annotation model. For example, the difference between the first confidence level and the second confidence level can be used as the degree of confidence decay of the data annotation model.

[0126] In step 124 above, when the confidence level decay is greater than or equal to a preset decay threshold, it is determined that the data labeling model needs model fine-tuning. When the confidence level decay is less than the preset decay threshold, it is determined that the data labeling model does not need model fine-tuning. When the data labeling model needs model fine-tuning, step 118 above is executed; otherwise, step 118 above is not executed.

[0127] In some embodiments, the fine-tuning gain of the data annotation model can be obtained, where fine-tuning gain refers to the performance gain obtained from the previous fine-tuning of the model. Whether the data annotation model needs fine-tuning can be determined by comprehensively considering both model gain and confidence decay. For example, when the model gain is greater than or equal to a preset gain, a first preset decay threshold is used to determine whether fine-tuning is needed; when the model gain is less than the preset gain, a second preset decay threshold is used to determine whether fine-tuning is needed. The first preset decay threshold is less than the second preset decay threshold.

[0128] In this embodiment, the performance degradation of the model is dynamically perceived, thereby dynamically determining whether to perform model fine-tuning.

[0129] In some embodiments, the above method may further include: 126. Analyze historical annotation results that are below the preset requirements to identify annotation difficulties.

[0130] 128. Based on the marked difficulties, update the task complexity attributes in the task profile mode to obtain the updated task profile mode.

[0131] The updated task profile mode is used to generate task profiles for tasks to be assigned and labeled.

[0132] In the aforementioned 126 cases, for annotation results with quality below the preset requirements, an annotation difficulty analysis is first performed. Specifically, the system collects low-quality annotation samples and their corresponding contextual information, including original data (annotated object) characteristics (such as image blurriness, text ambiguity), annotator's competency profile (such as accuracy, domain knowledge score), operation behavior logs (such as number of modifications, dwell time), and quality inspection feedback. Through statistical analysis and pattern mining of this multi-source data, key factors leading to annotation errors or inconsistencies are identified, such as: target object size being too small, category boundaries being blurry, semantic ambiguity, and unclear labeling system. This process can be combined with rule judgment, cluster analysis, or machine learning models to automatically classify annotation difficulty types, forming a structured annotation difficulty framework.

[0133] In the above 128, based on the identified annotation difficulties, the attributes related to task complexity in the task profile mode are dynamically updated. That is, through difficulty analysis, the factors that lead to increased complexity can be identified, and these factors are added to the task profile mode as sub-dimensions under the dimension of task complexity.

[0134] In practical applications, if analysis reveals that frequent labeling errors occur in a certain type of image due to the "dense distribution of small targets," then "spatial density" will be added as a sub-dimension under the task complexity dimension to the task profiling mode. If text annotations lead to discrepancies due to semantic ambiguity, then "semantic ambiguity" will be added as a sub-dimension under the task complexity dimension to the task profiling mode.

[0135] The technical solution provided in this embodiment realizes closed-loop optimization from "result feedback" to "profile mode", which can improve the accuracy of task profile and provide more accurate input basis for subsequent task allocation.

[0136] The following is combined Figure 2 The annotation task allocation method provided in the embodiments of this specification is illustrated with examples. For instance... Figure 2 As shown, the method includes the following steps: 20. Task Acquisition.

[0137] Obtain multiple tasks to be labeled. These tasks can come from the same labeling project or from multiple different labeling projects.

[0138] 21. Preprocessing.

[0139] The purpose of preprocessing is to standardize and structure the annotation objects in the annotation task, so as to provide a unified input standard for subsequent processing.

[0140] Typically, annotation objects include modal data such as text, images, and videos. For different modalities like text, images, and videos, format parsing and feature extraction are performed separately. For example, semantic segmentation and entity recognition are performed on the text data in the annotation task to be assigned, and the segmentation and entity recognition results are added to the annotation task for reference during subsequent annotation. For the image data in the annotation task to be assigned, standardized size adjustment and noise filtering can be implemented. The video data in the annotation task to be assigned is decomposed into a series of keyframes.

[0141] Optionally, domain-adaptive algorithms can be used to improve data labelability. For example, medical images can be automatically enhanced to improve the contrast of lesion areas, and legal texts can be injected with clause correlation analysis.

[0142] 22. Task profile construction.

[0143] Following a pre-defined task profiling model, the tasks to be assigned for annotation are analyzed to obtain metrics across multiple dimensions. Based on these metrics, a task profile for the assigned task is generated. The task profiling model includes several pre-defined dimensions, such as domain type, task type, task complexity, required skill type, and task requirements. The task complexity dimension can be further subdivided into several sub-dimensions, determined based on feedback. Specifically, based on historical annotation results where quality fell below pre-defined requirements, annotation difficulties are analyzed, and the sub-dimensions under the task complexity dimension are updated accordingly. Updates include adding new sub-dimensions and deleting existing ones.

[0144] 23. Task allocation based on optimization models.

[0145] Based on task profiles, capability profiles of multiple annotators, and capability profiles of multiple data annotation models, a three-dimensional optimization model is constructed to determine the allocation relationship between the annotation tasks to be assigned, the annotators, and the data annotation models. This model is used to simultaneously optimize three performance indicators: quality, cost, and timeliness. The adaptive NSGA-II algorithm is used to solve the three-dimensional optimization model to obtain the Pareto front. The Pareto front includes multiple solutions, which represent multiple alternative task allocation schemes. Based on the weights corresponding to the multiple performance indicators and the index values ​​of the multiple alternative task allocation schemes under each performance indicator, the performance scores of the multiple alternative task allocation schemes are determined. Based on the performance scores, the target task allocation scheme is selected from the multiple alternative task allocation schemes.

[0146] Among them, the Monte Carlo Tree Search (MCTS) algorithm can be introduced to select the target task allocation scheme from multiple alternative task allocation schemes.

[0147] The weights corresponding to various performance indicators are obtained by a dynamic weight adjuster. The dynamic weight adjuster determines the weights of various performance indicators based on the task labeling quality of processed tasks, the resource consumption information of processed tasks, and / or the time limits of the multiple unassigned labeling tasks.

[0148] Tasks are assigned according to the target task allocation plan. This allocation plan includes three task processing modes: manual mode, automatic mode, and human-machine collaborative mode.

[0149] 24. Manual annotation.

[0150] The annotators manually perform the annotation tasks on the terminal.

[0151] 25. Model annotation.

[0152] The data labeling model labels the assigned tasks and outputs the labeling results.

[0153] 26. Human-computer collaborative annotation.

[0154] For example, the large model generates initial labeling results, and the labelers make fine-tuning corrections based on these results. The system provides auxiliary suggestions in real time (such as highlighting differences and compliance checks).

[0155] After a task is assigned, the status of resources (personnel or models) can be monitored, and task migration or priority reordering can be automatically triggered when an anomaly occurs.

[0156] 27. Stratified quality verification.

[0157] Layered quality verification involves three layers of verification: basic verification layer, cross-validation layer, and arbitration decision layer.

[0158] For specific verification methods, please refer to the corresponding content in the above embodiments, which will not be repeated here.

[0159] 28. Arbitration Amendment.

[0160] The poor quality annotation results identified during the stratified quality verification process can be provided to experts for correction.

[0161] 29. Labeled output.

[0162] Output the final annotation results.

[0163] 31. Model fine-tuning.

[0164] It can detect the degree of confidence decay of the model in real time. When the confidence decay of the model is large, it can use contrastive learning algorithms and LoRA fine-tuning technology to fine-tune the model, so that the model can absorb the logical consistency and domain specificity of human annotations. The model continuously adapts to new scenarios as the labeled data accumulates, and the long-term performance stability is enhanced. Human expert experience is encoded into the model parameters, reducing the dependence on repeated annotations. The annotation error rate decreases iteratively in rounds, and the output consistency is significantly improved, realizing a closed-loop evolution from annotation results to model capabilities.

[0165] 32. Portrait update.

[0166] The profiles of the annotators and the profiles of the data annotation models are updated. For specific update methods and timing, please refer to the relevant content in the above embodiments, which will not be repeated here.

[0167] The technical solutions provided in the embodiments of this specification can enhance the balance between quality, cost, and timeliness; optimize resource utilization and avoid local overload or idleness; improve task processing timeliness (especially for urgent tasks); and enhance system robustness. Furthermore, the closed-loop evolution mechanism achieves continuous model optimization and continuous optimization of task profile dimensions through a self-feedback link.

[0168] One embodiment of this specification also provides a task assignment device, comprising: The first acquisition module is used to acquire multiple annotation tasks to be assigned. The determination module is used to determine the task profile of the multiple assignment tasks based on the multiple assignment tasks; The second acquisition module is used to acquire the capability profiles of multiple annotators and multiple data annotation models, wherein the data annotation models are used for data annotation. The construction module is used to construct an optimization model based on the task profile, the capability profiles of the multiple annotators, and the capability profiles of the multiple data annotation models to determine the allocation relationship between the annotation tasks to be assigned, the annotators, and the data annotation models. The solution module is used to solve the optimization model using a solution algorithm to generate a target task allocation scheme.

[0169] It should be noted that the devices provided in the above embodiments can implement the technical solutions described in the corresponding method embodiments above. The specific implementation principles and corresponding beneficial effects of the above modules or units can be found in the corresponding content of the above method embodiments, and will not be repeated here.

[0170] This specification also provides an electronic device according to one embodiment. For example... Figure 3 As shown, the electronic device includes a processor 42 and a memory 41. The memory 41 stores one or more computer programs (or instructions); the processor 42 is coupled to the memory 41 and is used for the at least one or more computer programs to implement the steps in the methods provided in the embodiments of this specification.

[0171] Furthermore, the electronic device also includes other components such as a communication component 43, a display 44, a power supply component 45, and an audio component 46. Only some components are shown here for illustrative purposes, and it is not intended that the electronic device contains only these components.

[0172] The methods in this specification can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. When implemented in software, they can be implemented, in whole or in part, as a computer program product. Therefore, this specification also provides a computer program product. This computer program product includes a computer program / instructions that, when executed by an electronic component such as a processor, can perform, in whole or in part, the steps or functions of the methods provided in the embodiments of this specification. The computer can be a general-purpose computer, a special-purpose computer, a computer network, network equipment, user equipment, core network equipment, or other programmable device.

[0173] The aforementioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0174] The aforementioned display includes a screen, which may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen can be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation.

[0175] The aforementioned power supply components provide power to various components within the device in which they reside. These power supply components may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device in which they reside.

[0176] The aforementioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, or voice recognition mode. The received audio signals can be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0177] Accordingly, embodiments of this specification also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the above-described method embodiments. The computer-readable storage medium includes volatile or non-volatile components, or a combination thereof, and can be removable or non-removable. Examples of computer-readable storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), flash memory or other memory technologies, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium.

[0178] Accordingly, embodiments of this specification also provide a computer program product, which includes a computer program or instructions that, when executed by a processor, cause the processor to implement the steps in the above-described method embodiments. It should be understood that each step or combination of steps in the above-described method flow can be implemented by the computer program or instructions. Furthermore, these computer programs or instructions can be applied to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device, enabling the processor of the general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to function as an apparatus for implementing the corresponding functions in the above-described method embodiments.

[0179] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0180] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for embodiments such as devices, electronic devices, storage media, and program products, since they are basically similar to the method embodiments, the descriptions are relatively simple, and relevant parts can be referred to the descriptions of the method embodiments.

[0181] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0182] The above are merely embodiments of this specification and are not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this application.

Claims

1. A method for assigning annotation tasks, characterized in that, The method comprises the following steps: obtaining a plurality of to-be-assigned annotation tasks; determining a task profile of the plurality of to-be-assigned annotation tasks according to the plurality of to-be-assigned annotation tasks; obtaining a plurality of ability profiles of annotation personnel and a plurality of ability profiles of data annotation models, the data annotation models being used for data annotation; constructing an optimization model for determining an assignment relationship between to-be-assigned annotation tasks, annotation personnel and data annotation models according to the task profile, the plurality of ability profiles of annotation personnel and the plurality of ability profiles of data annotation models; solving the optimization model by using a solving algorithm to generate a target task assignment scheme.

2. The method of claim 1, wherein, The solving algorithm comprises a genetic algorithm; solving the optimization model by using a solving algorithm to obtain a target task assignment scheme, comprising: encoding the solution of the optimization model to obtain an encoded solution, the encoded solution comprising a gene unit constructed for each to-be-assigned annotation task, the gene unit comprising: a mode selection gene, a personnel selection gene and a model selection gene, the mode selection gene being used to indicate a processing mode of the to-be-assigned annotation task, the personnel selection gene being used to indicate an assigned annotation personnel when the processing mode indicates that personnel is needed to participate, and the model selection gene being used to indicate an assigned data annotation model when the processing mode indicates that a data annotation model is needed to participate; based on the encoded solution, iteratively searching by using the genetic algorithm to generate a target task assignment scheme.

3. The method of claim 2, wherein, The processing mode comprises at least two of the following three modes: a manual mode in which annotation personnel performs annotation tasks without the participation of a data annotation model, an automatic mode in which a data annotation model performs annotation tasks without the participation of annotation personnel, and a man-machine collaborative mode in which annotation personnel and a data annotation model jointly participate in annotation tasks.

4. The method according to any one of claims 1 to 3, characterized in that, The optimization model is used to optimize a plurality of performance indicators; The plurality of performance indicators comprises at least two of annotation cost, annotation time and annotation quality.

5. The method of claim 4, wherein, solving the optimization model by using a solving algorithm to generate a target task assignment scheme, comprising: solving the optimization model by using a solving algorithm to obtain a Pareto solution set, the Pareto solution set comprising a plurality of solutions for representing a plurality of alternative task assignment schemes; determining the target task assignment scheme from the plurality of alternative task assignment schemes.

6. The method of claim 5, wherein, determining the target task assignment scheme from the plurality of alternative task assignment schemes, comprising: determining weights corresponding to the plurality of performance indicators based on task annotation quality of processed tasks, resource consumption information of processed tasks and / or time limits of the plurality of to-be-assigned annotation tasks; determining performance scores of the plurality of alternative task assignment schemes according to the weights corresponding to the plurality of performance indicators and index values of the plurality of alternative task assignment schemes under each performance indicator; selecting a target task assignment scheme from the plurality of alternative task assignment schemes according to the performance scores.

7. The method according to any one of claims 1 to 3, characterized in that, The ability profile of the data annotation model comprises at least one of the following attributes: domain ability, annotation quality, annotation efficiency, service stability, resource consumption cost, knowledge timeliness and confidence decay degree.

8. The method according to any one of claims 1 to 3, characterized in that, The method further comprises the following steps: According to the labeling data of the data labeling model, the capability profile of the data labeling model is updated.

9. The method according to any one of claims 1 to 3, characterized in that, Further comprising: Based on the historical labeling results of the labeling personnel and / or the historical labeling results of the data labeling model, training data is constructed, the training data including positive samples and negative samples; According to the training data, the data labeling model is fine-tuned through a contrastive learning algorithm.

10. The method of claim 9, wherein, Further comprising: After the data labeling model is fine-tuned, the capability profile of the data labeling model is updated according to the model fine-tuning result.

11. The method of claim 9, wherein, Further comprising: Obtain quality check information of the labeling result of the data labeling model; According to the quality check information, the confidence decay degree of the data labeling model is determined; According to the confidence decay degree, it is determined whether the data labeling model needs to be fine-tuned; According to the training data, the data labeling model is fine-tuned through a contrastive learning algorithm, including: When the data labeling model needs to be fine-tuned, the data labeling model is fine-tuned through a contrastive learning algorithm according to the training data.

12. The method of any one of claims 1 to 3, wherein, Further comprising: Analyze the labeling results with historical labeling quality lower than the preset requirement to determine the labeling difficulty; According to the labeling difficulty, update the attributes related to task complexity in the task profile mode to obtain an updated task profile mode; The updated task profile mode is used to generate a task profile of a to-be-assigned labeling task.

13. The method of any one of claims 1 to 3, wherein, According to the task profile, the capability profiles of the multiple labeling personnel and the multiple data labeling models, an optimization model is constructed, including: According to the multiple to-be-assigned labeling tasks, the multiple labeling personnel and the multiple data labeling models, a decision variable is determined, the decision variable being used to represent the allocation relationship between the to-be-assigned labeling tasks, the labeling personnel and the data labeling models; According to the task profile, the capability profiles of the multiple labeling personnel and the multiple data labeling models, a performance evaluation parameter is generated; According to the performance evaluation parameter and the decision variable, a target function is constructed; Based on the target function, an optimization model is constructed, with the goal of optimizing the target function.

14. An annotated task assignment apparatus characterized by comprising: Including: A first obtaining module is configured to obtain a plurality of to-be-assigned labeling tasks; A determining module is configured to determine a task profile of the plurality of to-be-assigned labeling tasks according to the plurality of to-be-assigned labeling tasks; A second obtaining module is configured to obtain capability profiles of a plurality of labeling personnel and capability profiles of a plurality of data labeling models, the data labeling models being used for data labeling; A constructing module is configured to construct an optimization model for determining an allocation relationship between to-be-assigned labeling tasks, labeling personnel and data labeling models according to the task profile, the capability profiles of the plurality of labeling personnel and the capability profiles of the plurality of data labeling models; A solving module is configured to solve the optimization model by using a solving algorithm to generate a target task allocation scheme.

15. An electronic device, comprising: Including: A memory and a processor, wherein The memory is configured to store a program; The processor, coupled with the memory, is configured to execute the program stored in the memory to implement the method in any one of claims 1 to 13.

16. A computer readable storage medium storing a computer program, characterized in that, The computer program, when executed by a computer, is capable of implementing the method in any one of claims 1 to 13.

17. A computer program product comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the method in any one of claims 1 to 13.

Images