Team capability optimization method, system and device, electronic equipment, storage medium and program product

Abstract

The embodiment of the invention provides a team capability optimization method, system and device, electronic equipment, a storage medium and a program product. According to the scheme provided by the embodiment of the invention, after the task execution behavior log of each member in the team is analyzed and the task execution behavior characteristics capable of reflecting the work task execution capability of the member are extracted, the capability score of each member is calculated based on the task execution behavior characteristics of each member; and determining a score threshold according to the ability score of each member, and screening out an adaptive member as a task expert according to the score threshold and the ability score of each member. Furthermore, capability optimization is carried out on the team according to related task behavior information of the task expert determined according to the task execution behavior characteristics of the task expert and the task execution behavior log and the current team capability state. The capability optimization comprises at least one of the following steps: pushing a task execution training scheme to the members, and adjusting a task cooperation relationship among the members.

Description

Technical Field

[0001] This specification relates to the field of data annotation, and in particular to a method, system, device, electronic device, storage medium, and program product for optimizing team capabilities. Background Technology

[0002] The rapid development of artificial intelligence (AI) technology has led to the widespread application of large-scale models in fields such as computer vision, natural language processing, and speech recognition. The performance of large-scale models heavily relies on massive amounts of high-quality labeled data as a training foundation to learn complex feature representations and semantic patterns. In practical applications, data labeling is typically completed by specialized labeling teams. As AI technology applications continue to deepen, the requirements for the accuracy and diversity of training data for large-scale models have significantly increased. Data standardization tasks are exhibiting characteristics of high complexity, strong professionalism, and dynamic evolution. In this context, the capabilities of labeling teams are no longer limited to the accuracy and efficiency of individual labeling, but extend to team-level knowledge integration capabilities, consensus-building mechanisms, and dynamic adaptability to task evolution.

[0003] Therefore, in the face of increasingly complex data annotation needs, further exploring technical solutions that can promote the continuous evolution of annotation team capabilities has become a key direction for promoting the large-scale production of high-quality training data. Summary of the Invention

[0004] Several embodiments in this specification provide a method, system, apparatus, electronic device, storage medium, and program product for optimizing team capabilities, enabling the continuous evolution of data annotation team capabilities. Among them, In a first embodiment, this specification provides a method for optimizing team capabilities. The method includes: Obtain the task execution behavior characteristics of each member in the team; the task execution behavior characteristics are extracted by analyzing the task execution behavior logs of the members and can reflect the members' work task execution capabilities; Based on the aforementioned task execution behavior characteristics, calculate the ability score of each member; Determine the score threshold based on each member's ability score; Based on the scoring threshold and the ability scores of each member, a member screening operation is performed to identify the screened members as task experts; Based on the task execution behavior characteristics and task execution behavior log of the task expert, the relevant task behavior information of the task expert is determined; Based on the relevant task behavior information of the task experts and the current team capability status, the team's capabilities are optimized; wherein, the capability optimization includes at least one of the following: pushing task execution training programs to members to compensate for the team's shortcomings in task execution capabilities, and adjusting the task collaboration relationships among members.

[0005] In a second embodiment, this specification provides a team capability optimization system. The system includes: The client is used to display the client interface and collect the actions of each member of the team in performing work tasks through the client interface, and send the actions to the server. The server is used to record the received behavioral operations in the corresponding member's task execution behavior log; and is also used to implement the method provided in the first embodiment above.

[0006] In a third embodiment, this specification provides a team capability optimization device. The device includes: The acquisition module is used to acquire the task execution behavior characteristics of each member in the team; the task execution behavior characteristics are extracted by analyzing the task execution behavior logs of the members and can reflect the members' work task execution capabilities; The calculation and determination module is used to calculate the ability score of each member based on the task execution behavior characteristics; and to determine the score threshold based on the ability score of each member. The execution module is used to perform a member screening operation based on the scoring threshold and the ability scores of each member to determine the screened members as task experts. The calculation and determination module is further configured to determine the relevant task behavior information of the task expert based on the task execution behavior characteristics and the task execution behavior log of the task expert; The optimization module is used to optimize the capabilities of the team based on the relevant task behavior information of the task expert and the current team capability status of the team; wherein, the capability optimization includes at least one of the following: pushing task execution training plans to members and adjusting the task collaboration relationships between members.

[0007] In a fourth embodiment, this specification provides an electronic device including a memory and a processor, wherein the memory stores executable program instructions, and when the processor executes the program instructions, it implements the method provided in the first embodiment.

[0008] In a fifth embodiment, this specification provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed in a computer, it causes the computer to perform the method provided in the first embodiment.

[0009] In a sixth embodiment, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the method provided in the first embodiment.

[0010] The solutions provided in the above embodiments of this specification, after analyzing the task execution behavior logs of each member in the team and extracting task execution behavior characteristics that reflect the member's work task (such as data annotation task) execution ability, calculate the ability score of each member based on the task execution behavior characteristics (such as annotation behavior characteristics), and determine a score threshold based on the ability score of each member, thereby enabling the selection of suitable members as task experts based on the score threshold and the ability scores of each member. Here, a dynamic threshold screening method is implemented to screen task experts (such as annotation experts), which can better adapt to changes in task complexity and avoid the fixation of task experts and misjudgment. Furthermore, after determining the relevant task behavior information of task experts based on the task execution behavior characteristics and task execution behavior logs, the team's capabilities are optimized based on the relevant task behavior information of task experts and the current team capability status. The capability optimization mentioned here includes at least one of the following: pushing task execution training programs to members and adjusting the task collaboration relationships between members. When performing team capability optimization, this solution will push task execution training programs to members to address the team's task execution capability shortcomings, which realizes targeted training for the team. In addition, by dynamically adjusting the collaborative relationships among members, a qualitative change can be achieved in the team from "the distribution of individual capabilities" to "the emergence of collective wisdom", thereby improving the rationality of task allocation and the efficiency of group consensus. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the various embodiments disclosed in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely examples of the various embodiments disclosed in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort. In the drawings: Figure 1 A schematic diagram of the technical architecture on which the method implementation in this specification is based, provided for exemplary embodiments; Figure 2 A flowchart illustrating the optimization of data annotation team capabilities provided for exemplary embodiments in this specification; Figure 3 A schematic diagram of the structure of a team capability optimization system provided as an exemplary embodiment of this specification; Figure 4 A flowchart illustrating the team capability optimization method provided for exemplary embodiments of this specification; Figure 5 A schematic diagram of the team capability optimization device is provided for the exemplary embodiments described in this specification; Figure 6 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this specification. Detailed Implementation

[0012] As mentioned earlier, with the rapid development of artificial intelligence technology, the demand for high-quality training data is also increasing. Training data is usually obtained by professional annotation teams through semantic parsing and structured annotation of raw data. This annotated data is used for training large models and directly determines the learning direction and performance ceiling of the large models. Faced with the increasingly complex demand for high-quality training data, higher requirements are placed on the capabilities of data annotation teams. However, the current management of data annotation teams has the following significant limitations: low knowledge transfer efficiency, lagging group decision-making, and lack of dynamic adaptability. Low knowledge transfer efficiency means that the experience of annotation experts is often limited to the individual level, and there is a lack of effective knowledge sharing mechanisms within the team, making it difficult to transform high-quality experience into group consensus. Lagging group decision-making means that the existing annotation process usually relies on centralized manual review, which is lengthy and inefficient in terms of error correction, and cannot meet the needs of rapid iteration of large models. Lack of dynamic adaptability means that it is difficult to capture the evolution of the capabilities of the data annotation team and predict the direction of the migration of group cognitive patterns, resulting in stagnation of team capabilities. The technical factors that cause the above limitations in the current management of data annotation teams include: shallow application of swarm intelligence, static knowledge management mechanisms, and the separation between individuals and groups. Superficial swarm intelligence applications refer to situations where most data annotation team management solutions only employ simple voting or majority voting strategies, failing to delve into the correlations in decision-making patterns within annotator behavior data and ignoring the deeper patterns in group collaboration. Static knowledge management mechanisms refer to situations where annotation rule base updates rely on manual intervention, lacking dynamic linkage with the evolution of large model parameters, leading to a disconnect between annotation rules and the capabilities of the large model. The separation between individuals and the group refers to the failure to establish a two-way reinforcement loop between expert experience and collective wisdom; team capability improvement primarily relies on external intervention rather than self-organizing mechanisms to achieve capability evolution.

[0013] For example, in relevant data annotation team management technical solutions, one approach uses traditional swarm intelligence algorithms (such as majority voting and weighted average). This approach is characterized by its application scenarios being crowdsourced annotation tasks (such as image classification and text annotation), and its core mechanism generating group decisions through simple aggregation of annotation results (such as voting and averages). However, it suffers from drawbacks such as shallow decision modeling, rigid weight allocation, and weak anti-interference capabilities. Shallow decision modeling refers to relying solely on the final annotation result while ignoring behavioral characteristics during the annotation process (such as response latency and rule application patterns). Rigid weight allocation means that traditional weighted algorithms (such as those based on historical accuracy) cannot dynamically adapt to changes in task complexity, resulting in limited quality of group consensus. Weak anti-interference capabilities mean that there is a lack of identification and filtering mechanisms for noisy annotators or malicious attacks, making group decisions susceptible to outliers. Another approach uses static knowledge management solutions (such as manual annotation rule bases). This approach is characterized by its application scenarios being specialized annotation systems in high-barrier fields such as medicine, and its core mechanism being that domain experts manually write annotation rules to form a fixed knowledge base for team reference. However, this static knowledge management solution suffers from problems such as delayed updates, implicit experience, and lack of personalization. Delayed updates mean that the rule base relies on manual iteration, making it difficult to keep pace with the rapid expansion of large model capabilities (such as new entity types and fuzzy semantic scenarios). Implicit experience means that experts' tacit knowledge (such as intuitive judgments and edge case handling skills) is difficult to store in a structured way, resulting in low reusability. Lack of personalization means that a unified rule base cannot adapt to the cognitive differences of different annotators, leading to inconsistent execution efficiency.

[0014] To address the aforementioned issues, this specification proposes a data annotation team capability evolution scheme based on expert behavior analysis. This scheme aims to overcome the capability bottlenecks of traditional data annotation teams by dynamically identifying high-value annotators, distilling their decision-making patterns, and driving the optimization of the group collaboration network. Specifically, the basic idea of ​​this scheme includes: constructing a behavior analysis model to accurately identify experts from dimensions such as the completeness and quality of annotation tasks, temporal stability, and rule transferability; making expert implicit experience explicit into reusable group knowledge assets through adversarial example generation and knowledge graph modeling; and dynamically reconstructing the collaboration network topology to achieve a qualitative leap in the data annotation team from "discrete distribution of individual capabilities" to "emergent collective wisdom." This scheme effectively resolves the contradictions in existing data annotation team management technologies, such as the difficulty in replicating expert experience, lagging group decision-making, and the inability of team capabilities to adapt to large model iterations. It provides a closed-loop solution for continuously pre-training large models with high-quality data.

[0015] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments in this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0016] It should be noted that, for ease of description, the accompanying drawings only show the parts related to the relevant technical solutions. Unless otherwise specified, the embodiments and features described in this specification can be combined with each other. Furthermore, the terms "first," "second," and "third" used in the embodiments of this specification are for informational purposes only and do not constitute any limitation. Moreover, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitations, the presence of other identical or equivalent elements in the process, method, product, or apparatus that includes the stated elements is not excluded. Furthermore, in this specification, unless explicitly stated otherwise, "receiving and transmitting data" does not necessarily mean direct receiving and transmitting; it can be indirect receiving and transmitting. For example, when A receives data sent by B, it can be understood as A directly receiving the data sent by B, or it can be understood as A indirectly receiving the data sent by B through other entities such as C. Similarly, when B sends data to A, it can be understood as B sending the data directly to A, or it can be understood as B indirectly sending the data to A through other entities such as C. Here, C can be one entity, or it can be two or more entities.

[0017] Furthermore, it should be noted that specific terms are used to describe embodiments of this specification. For example, "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic related to at least one embodiment of this specification. Therefore, it should be emphasized and noted that "an embodiment," "one embodiment," or "an alternative embodiment" mentioned twice or more in different locations in this specification do not necessarily refer to the same embodiment. Moreover, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples, without contradiction. Although one or more embodiments of this specification provide method steps as described in the embodiments or flowcharts, it is understood that the order of steps listed in the embodiments or flowcharts is merely one possible order of execution among many steps, and does not represent the only possible order. Therefore, when the claims involve method steps, adjustments to the order of such steps, or parallel execution between steps, are also within the scope of protection of the claims.

[0018] Furthermore, it should be noted that the user data obtained in this manual is authorized by the user and does not involve user privacy.

[0019] The embodiments provided in this specification will be described below with reference to the accompanying drawings.

[0020] First, the terminology used in the embodiments of this specification will be explained. It should be understood that this explanation is for the purpose of providing a clearer understanding of the embodiments described herein and does not necessarily constitute a limitation on the embodiments of this specification.

[0021] The expert feature modeling module is defined as follows: It constructs a multi-dimensional feature vector space by collecting non-biological behavioral data on annotators, such as task completion quality, response time stability, rule application consistency, cross-task transferability, and knowledge dissemination efficiency. Its function is to screen high-value experts based on dynamic threshold models (such as the statistical mean ± k standard deviations) and analyze the interpretability characteristics of their decision-making patterns.

[0022] A dynamic threshold model is defined as an algorithmic model that dynamically adjusts the expert recognition threshold based on the historical behavioral data of annotators performing data annotation tasks. For example, a statistical method based on a sliding window to calculate the mean and variance can be used to automatically determine the expert recognition threshold using the mean ± k times the standard deviation. Here, the annotation difference is the arithmetic square root of the variance, and k can take values ​​such as 1 or 2. Its function is to adapt to the expert screening needs of different task scenarios and avoid misjudgments caused by fixed thresholds.

[0023] An experience distillation engine is defined as a processing unit that transforms the implicit decision-making experience of annotation experts into a structured knowledge base. It includes functions such as decision tree modeling, adversarial example generation, and annotation experience knowledge graph construction. Its role is to generate adversarial examples by simulating complex annotation task scenarios that annotation experts excel at handling through adversarial generative networks, and to generate a reusable annotation experience knowledge graph by combining the decision paths of annotation experts.

[0024] Generative adversarial examples are defined as follows: using Generative Adversarial Networks (GANs) to simulate complex data labeling tasks (such as boundary labeling), generating more challenging labeling task samples to test the decision boundaries of experts and the group. Their function is to enhance the robustness of labeling experience knowledge graphs and expose potential vulnerabilities in labeling rules.

[0025] Boundary labeling tasks (also known as fuzzy labeling tasks) are data labeling tasks that are "seemingly correct but actually ambiguous" and have uncertainties or ambiguities in semantic understanding, category attribution, boundary determination, or the application of labeling rules. This often leads to inconsistencies and disputes in the labeling results of different labelers or the same labeler at different times.

[0026] Annotation experience knowledge graph is defined as a semantic network that stores annotation rules, error patterns, and correction logic in a structured manner, including entities (such as annotation rules and case templates) and semantic relationships between entities (such as "cause ->" and "correction ->"). Its function is to enable explicit storage and group sharing of expert experience, and to support dynamic updates of annotation rules.

[0027] The team evolution system is defined as: a system that dynamically optimizes the collaboration mode of a data annotation team based on expert feature vectors, including personalized training program generation, collaborative network topology reconstruction, and real-time feedback control mechanisms. Its function is to drive continuous iteration of group annotation capabilities, improving the team's efficiency in handling complex cases and enhancing decision-making consistency.

[0028] Collaborative network topology reconstruction is defined as a mechanism for dynamically adjusting team collaboration relationships based on the similarity of annotators' abilities. It calculates the association weights between annotators using methods such as Jaccard similarity or cosine distance to determine their collaborative relationships. Its function is to optimize information transmission paths and improve the efficiency of group decision-making and the effectiveness of knowledge dissemination.

[0029] The group consistency index is defined as the degree of similarity among the annotation results of various annotators in a data annotation team for the same batch of annotation tasks (such as image classification, entity recognition, sentiment annotation, etc.). It can be measured, for example, by cosine similarity or the Kappa coefficient. Its function is to reflect the level of team collaboration and serve as a key indicator for triggering mechanisms such as expert experience broadcasting.

[0030] Decision path analysis is defined as: parsing the complete sequence of actions performed by annotators from receiving annotation results from a data annotation task, and extracting key decision nodes and annotation rule application patterns. Its function is to reveal the behavioral differences between annotation experts and ordinary annotators, providing input for the experience distillation engine.

[0031] Real-time feedback controller: Based on real-time calculated group annotation quality metrics (such as the group consistency index), it dynamically triggers intervention mechanisms (such as expert experience sharing and collaboration network adjustments). Its function is to ensure that the data annotation team's capabilities adapt quickly to changes in data annotation task requirements, avoiding decision-making lag. In this specification, "real-time" does not mean "instantaneous completion" or "zero latency," but rather refers to a rapid response and calculation within an acceptable time window (such as every minute) and / or event window (such as every 10 data annotation tasks completed). For example, calculating the Kappa value of the 10 most recent data annotation tasks every 30 seconds.

[0032] Large models refer to artificial intelligence models with a huge number of parameters and complex structures. These models typically consist of millions or even billions of parameters and can perform complex tasks such as language understanding and natural language processing. In this specification, a large model may be, for example, a language model (LLM).

[0033] The technical solutions provided in the embodiments described below are all based on Figure 1 The technical architecture shown is illustrated below. Taking the optimization of the data annotation team's capabilities as an example, this technical architecture aims to dynamically identify high-value experts within the data annotation team by analyzing non-biological behavioral data of each annotator, distilling expert decision-making patterns, and driving the dynamic optimization of the group collaboration network, thereby improving the overall annotation quality and consistency of the data annotation team. The core idea of ​​this technical architecture is to construct a closed-loop system of "identification-distillation-team evolution" to solve the problems of isolated expert experience, lagging group decision-making, and poor dynamic adaptability in data annotation teams.

[0034] like Figure 1 As shown, this technical architecture includes several functional modules: 1) Data Acquisition Module 11 The inputs to the data acquisition module 11 include: data annotation tasks and annotation holographic data generated by each annotator in the data annotation team during the execution of data annotation tasks. The data annotation tasks include the raw data to be annotated, which includes, but is not limited to, one or more modalities such as text, images, audio, video, and point cloud data. The annotation holographic data includes annotation process behavior data and annotation results. Annotation process behavior data refers to the behavioral data generated by the annotator from receiving the data annotation task to obtaining the annotation result, such as the total time taken for the data annotation task (also known as task completion time), annotation rule application records, and error correction paths. The error correction path refers to the sequence of actions and decision-making trajectories taken by the annotator from receiving feedback to completing the correction after the submitted annotation result has been reviewed, returned, or marked as erroneous. For example, the error correction path records information such as the repair time and the repair strategy. The annotation results are the labels used to annotate the raw data, such as bounding boxes and classification labels.

[0035] Furthermore, the output of the data acquisition module 11 is the annotation behavior log of each annotator. The annotation behavior log records the annotation process behavior data generated by the annotators during the execution of the data annotation task, as well as the annotation results. In addition to the above, in order to avoid the annotation behavior log losing its semantics, the annotation behavior log may also include some relevant information about the original data, such as the unique identifier of the original data, data type, annotation difficulty, etc.

[0036] 2) Expert Feature Modeling Module 12 The input to the expert feature modeling module 12 is the annotation behavior log of each annotator output by the data acquisition module 11. Furthermore, the expert feature modeling module 12 performs the following processes on the input annotation behavior log: a. Extracting multi-dimensional annotation behavior features: By parsing and analyzing the annotation behavior logs, the following dimensions of annotation behavior features that can reflect the annotation ability of the annotator can be extracted: data annotation task completion quality (which can be referred to as "annotation quality"), response time stability, consistency of annotation rule application, cross-task transfer ability, knowledge dissemination efficiency, etc.

[0037] In the above, the quality of data annotation task completion refers to the accuracy, completeness, and standardization of annotations performed by the annotator. Response time stability refers to the degree of fluctuation in the time consumed by the annotator when completing similar data annotation tasks. Consistency in annotation rule application refers to whether the annotator maintains consistency in applying the same or similar annotation rules across different annotation tasks or at different times. Knowledge dissemination efficiency refers to the speed and accuracy with which annotators internalize and stably apply new annotation rules or annotation training content to actual data annotation tasks after their release, reflecting the annotator's learning ability and rule adaptability. Cross-task transferability refers to the annotator's ability to effectively apply the annotation rule understanding, judgment logic, or cognitive strategies mastered in one type of data annotation task (such as text annotation) to other different types of data annotation tasks (such as image annotation, audio annotation, video annotation, etc.), reflecting the annotator's general cognitive level.

[0038] b. Based on the extracted multidimensional annotation behavior features, a dynamic threshold model is used to automatically determine the score threshold (i.e., the aforementioned expert identification threshold) to filter out the top N (TopN) annotators with the highest ability scores and / or those whose ability scores exceed the score threshold. These selected annotators are marked as high-value annotation experts. N is a positive integer.

[0039] The dynamic threshold model can be, but is not limited to, based on statistics. For example, in this dynamic threshold model, the dynamic threshold T = mean. 2 times the standard deviation (i.e.) ).

[0040] For example, assuming there are 20 data labelers in the data labeling team, a weighted fusion method can be used to generate a comprehensive ability score for each labeler based on their multidimensional behavioral characteristics. For instance, the comprehensive ability score for each labeler can be calculated using the following formula: ,in, These can represent different dimensions of behavioral characteristics, such as the quality of data annotation task completion, response time stability, consistency of rule application, and cross-task transferability. `w` represents the weight of the corresponding behavioral characteristic, and these weights can be flexibly adjusted according to actual circumstances. For example, weights can be dynamically adjusted using statistical methods (such as Principal Component Analysis (PCA)) or learning methods (such as learning to rank). After calculating the comprehensive ability score for each annotator, statistical analysis can be performed on the comprehensive ability scores to determine the corresponding average score. and score standard deviation Next, based on the average score... and score standard deviation According to the pre-set dynamic threshold setting strategy (such as using...) (As a scoring threshold), a scoring threshold was determined. Then, from the 20 annotators, those with a comprehensive ability score greater than or equal to the threshold were selected. The N annotators who meet this scoring threshold and rank highest in overall ability are considered high-value annotation experts.

[0041] This solution utilizes a dynamic threshold screening method based on the multidimensional behavioral characteristics of annotators to select annotation experts. This approach better adapts to changes in the complexity of data annotation tasks and avoids the fixation of annotation experts and misjudgments. Moreover, the annotation expert selection process does not rely on biometric data collection (such as eye tracking), reducing hardware costs and privacy risks.

[0042] c. Output the expert feature fingerprint database. This database stores the multi-dimensional behavioral features of each annotation expert and their annotation process behavior data.

[0043] Among them, the multidimensional behavioral characteristics of annotation experts refer to the multidimensional behavioral characteristics used in the selection of annotation experts, such as the quality of annotation task completion and the stability of response timing.

[0044] Furthermore, the annotation process behavior data can be obtained from the annotation expert's annotation behavior logs, which may contain the annotation expert's decision-making path information. Understandably, decision-making path information refers to the complete logical chain and sequence of cognitive judgments and operational behaviors of the annotation expert when completing a data annotation task. It not only records "what was ultimately annotated," but also reveals the internal reasoning process of "why it was annotated this way, and how the annotation result was obtained step by step," serving as a visible expression of the annotation expert's implicit experience. That is, the annotation expert's decision-making path information can refer to the ordered set of judgments, reasoning, verification, and corrections experienced by the annotation expert at key cognitive nodes from the receipt of the data annotation task to the submission of the annotation result, reflecting the annotation expert's cognitive logic and problem-solving strategies. For example, after receiving a data annotation task, an annotation expert might first check relevant annotation rules (such as emotion recognition specifications), data annotation task examples, etc., and then execute the data annotation task based on this information. This expert experience of checking before performing the data annotation task is also stored in the expert feature fingerprint database, specifically, as part of the decision-making path information.

[0045] In this specification, the decision path information includes, but is not limited to, the order in which annotation experts consult and reference annotation rules during the execution of data annotation tasks, the reference behavior of data annotation task examples, the annotation modification trajectory, the annotation reasoning logic (specifically how annotation is implemented), and the different decision-making sequences used for annotation (the order of the cognitive steps, operational procedures, and judgment logic adopted, such as sequentially performing annotation rule consultation, data standard task example reference, preliminary judgment, modification verification, and final confirmation of annotation results).

[0046] In the future, the aforementioned expert feature fingerprint database can be used to guide and optimize the capabilities of each annotator in the data annotation team.

[0047] 3) Experience Distillation Engine 13 The input to the experience distillation engine 13 includes the aforementioned expert feature fingerprint database, and its main function is to transform the implicit annotation experience knowledge of annotation experts into a quantifiable, interpretable, and reusable structured knowledge system through certain technical means (such as algorithm analysis).

[0048] In practice, the experience distillation engine 13 performs the following processing operations based on the input expert feature fingerprint database: a. Based on the constructed decision tree model, the expert decision path is parsed and labeled.

[0049] This can be understood as using decision tree models to analyze and interpret the logic followed by annotation experts when making decisions. Specifically, it involves using decision tree models to analyze the annotation process behavior data of each annotation expert stored in the expert feature fingerprint database, thereby identifying the decision logic and cognitive patterns of each annotation expert.

[0050] For example, the annotation process behavior data of annotation experts records various behaviors such as how annotation experts handle boundary annotation tasks, resolve annotation conflicts, and apply complex annotation rules. By using a decision tree model to conduct in-depth analysis of the annotation process behavior data, the various behaviors in the annotation process behavior data will be transformed into quantifiable annotation knowledge (also known as quantifiable features), such as annotation path (e.g., "check rules -> compare sample cases -> annotate -> add notes"), rule application logic (e.g., "if the sample is at night and has reflective areas, then do not annotate puddles"), and error correction strategy (e.g., "after the annotation is rejected after review -> check the annotation rules -> modify the annotation").

[0051] The "boundary labeling task" mentioned in the above example is also called a fuzzy labeling task. It refers to a data labeling task where, due to semantic ambiguity, incomplete information, vague boundaries, or the existence of multiple reasonable interpretations, it is difficult for the labeler to make a unique and clear judgment based on existing rules. For example, the data labeling task "Your service is really good, I'll come back to cause trouble next time" appears to be a positive statement, but it actually carries a negative sentiment.

[0052] b. Constructing a knowledge graph of labeled experience Here, the aforementioned annotation knowledge is structured to transform it into a storable, searchable, and disseminable annotation experience knowledge graph. This enables the explicit storage and reuse of annotation experts' annotation experience, making it a resource that annotation teams can learn from and apply, thereby supporting the improvement of annotation team capabilities.

[0053] In practice, one approach is to first generate corresponding annotation experience templates based on the annotation knowledge obtained from the annotation experts, such as generating a standard annotation process template based on the annotation path. These generated annotation experience templates are then stored centrally in an experience template library for centralized management, preventing them from being scattered. Furthermore, based on the continuously added annotation experience templates in the experience template library, an incremental update method can be used to construct an annotation experience knowledge graph.

[0054] Here, the approach of "first generating annotation experience templates, and then incrementally updating the annotation experience knowledge graph based on the annotation experience templates" is adopted. On the one hand, the annotation experience templates facilitate the improvement of knowledge (entities and entity relationships) extraction efficiency and the speed of incremental updates of the annotation experience knowledge graph. On the other hand, when it is necessary to trigger expert experience broadcasting in the future (such as when the group consistency index of the annotation team is detected to be lower than the preset threshold), it is convenient to push the appropriate annotation experience templates of annotation experts to the annotators.

[0055] The aforementioned annotation experience knowledge graph includes entities and relationships between them. Entities include, but are not limited to, annotation rules, typical cases, and error correction patterns. Relationships include, but are not limited to, causing ->, correcting ->, and avoiding ->. For example, the entities and relationships between entities included in the annotation experience knowledge graph include: <Annotation Rule 1.2> — Avoidance -> <Mislabeled emotion as complaint>, <Mislabeled waterhole> — Cause -> <Review and rework>, <Nighttime path judgment> — Applicable to -> <Low-light task>, <New sample> — Analogous to -> <Ironic standard example>, and so on.

[0056] This solution employs an incremental learning mechanism—updating the annotation experience knowledge graph incrementally—to achieve real-time absorption of new annotation rules. For example, when the large model adds the ability to process multilingual text, this technical architecture automatically selects annotation experts skilled in multilingual annotation, extracts their annotation experience, generates annotation rules adapted to the new requirements, and adds them to the annotation experience knowledge graph. This effectively achieves the synchronous evolution of the annotation experience knowledge graph and the capabilities of the large model.

[0057] Furthermore, in addition to the two processing operations mentioned above, the processing flow executed by the empirical distillation engine 13 may also include other processing operations, such as generating adversarial tasks to simulate complex data labeling tasks.

[0058] Specific implementation methods for generating adversarial tasks may include: based on an expert feature fingerprint database, identifying complex data annotation tasks that annotation experts are adept at handling, such as boundary labeling tasks (e.g., 60% occlusion, ironic tone, etc.). When identifying complex data annotation tasks that annotation experts are adept at handling, factors such as the difficulty of the data annotation task and the accuracy of the annotation results can be considered. Then, using Generative Adversarial Networks (GANs) or perturbation enhancement techniques, a simulated annotation task is generated to simulate the complex data annotation tasks that annotation experts are adept at handling; this simulated annotation task is the generated "adversarial task."

[0059] Here, the generated adversarial task can be specifically called an adversarial annotation task, or adversarial example.

[0060] For example, some boundary labeling tasks that annotation experts often deal with include "I've been waiting for almost a week and haven't received the goods yet, how can I return them?" By simulating this boundary labeling task, the generated adversarial task could be "The courier service hasn't been updated, can I only apply for a refund?"

[0061] The generated adversarial tasks can be pushed to various annotators in the data annotation team for annotation practice. Ordinary annotators in the data annotation team may disagree due to a lack of contextual understanding; for example, some ordinary annotators might classify the given adversarial task as "complaining about logistics," while others might classify it as "applying for a refund." High-value annotation experts in the data annotation team, based on their past annotation experience, determine that "when a user explicitly expresses 'applying for a refund,' even if logistics issues are mentioned, it should be prioritized as 'applying for a return / refund.'" The annotation expert's annotation judgment logic for this adversarial task is recorded, and the annotation rule for making this annotation decision is extracted: "If the statement contains explicit action words such as 'apply for a refund' or 'I want a refund,' it should be prioritized as 'applying for a return / refund,' regardless of whether logistics issues are mentioned." This annotation rule and adversarial task can be updated to the knowledge graph using an incremental update method.

[0062] This solution generates adversarial tasks to simulate complex data annotation tasks that annotation experts excel at handling. This allows the data annotation team to be trained in advance through adversarial tasks, improving their ability to process complex tasks and preventing annotation quality from lagging behind the large model's discovery. It also proactively exposes blind spots in the data annotation team's capabilities and reveals vulnerabilities in the annotation rules. These exposed issues can then be addressed through expert experience distillation, enhancing the robustness of the annotation team's collective decision-making.

[0063] 4) Team Evolution System 14 The core objective achieved through this annotation team evolution system 14 is to optimize the collaborative network of data annotation teams.

[0064] Furthermore, the inputs to the annotation team evolution system 14 include: the aforementioned annotation experience knowledge graph and the current team capability status of the data annotation team. This current team capability status is a group-level cognitive indicator, not a simple summation of individual capabilities, but a comprehensive representation of the overall behavioral patterns, capability distribution, and collaborative effectiveness of the same data annotation team within a specific time window. Specifically, the current team capability status can be defined as: the overall behavioral characteristics, capability level, cognitive consistency, annotation response time, and annotation accuracy exhibited by the data annotation team during the execution of data annotation tasks within the current time window.

[0065] In practice, the current team capability status mentioned above may be, but is not limited to, consisting of the following dimensions, each supported by multiple quantifiable behavioral characteristic indicators: annotation response efficiency (such as average annotation response time, annotation throughput, annotation task delay rate, etc.), overall annotation accuracy (such as average accuracy rate, review rework rate, etc.), cognitive consistency (annotation disagreement rate, etc.), and difficulty adaptability (such as the difference in annotation accuracy between simple and complex annotation tasks, and the success rate of handling fuzzy samples).

[0066] The Team Evolution System 14, based on input information (including knowledge graph and current team state), performs the following processing operations: a. Generate personalized training programs Here, we can first identify the weaknesses of the data annotation team (i.e., the team's shortcomings in annotation capabilities) based on the current capabilities of the team. Then, we can match appropriate annotation experts to address these weaknesses, and generate a personalized annotation training program based on the relevant annotation knowledge of the matched experts. The output form of this annotation training program can be, but is not limited to, micro-lecture videos, interactive exercises, etc.

[0067] For example, assuming that based on the current team status, it is identified that the data annotation team has a low accuracy rate in audio annotation, it indicates that audio annotation is a weak link in the team. In this case, an annotation expert who is good at audio annotation will be matched to address this weakness, and an audio annotation training program will be generated based on the audio annotation knowledge content related to the annotation expert stored in the annotation experience knowledge graph, so as to strengthen the data annotation team in audio annotation.

[0068] b. Dynamically reconstruct the collaborative network topology The core idea of ​​"dynamically reconstructing the collaborative network topology" here is to treat annotators as nodes in a collaborative network. The connecting lines (often called "edges") between different nodes represent the collaborative relationships between different annotators. However, the collaborative relationships between annotators are not fixed but are dynamically adjusted based on the annotator's annotation performance. Annotation performance can include the annotator's annotation process behavior data and annotation results during the execution of the annotation task.

[0069] In practice, similarity measurement methods (such as Jaccard similarity or cosine distance) can be used to quantify the consistency among annotators, determine the collaborative relationship between different annotators, and thus achieve dynamic adjustment of the collaborative network.

[0070] For example, suppose there are 20 annotators in the data annotation team. Initially, since these 20 annotators haven't performed any annotation tasks, they have no historical annotation performance data (such as historical annotation behavior data). In this case, data annotation tasks can be assigned to each annotator randomly or in a round-robin fashion. Further, after a period of time, the assignment of data annotation tasks to annotators can be paused, and the "building a collaborative network" phase can begin. During the collaborative network construction process, the Jaccard similarity between each pair of annotators can be calculated based on their historical annotation performance, and then the collaborative network is built based on these pairwise Jaccard similarities. In this process, each annotator is considered a node in the collaborative network. If the Jaccard similarity between two annotators is higher than a set threshold θ, an edge is established between the two nodes representing the two annotators, indicating a collaborative relationship between them. Subsequently, when the collaborative network adjustment trigger conditions are met, such as when the group consistency index of the data annotation team is detected as not meeting the standard, the collaborative network can be dynamically adjusted to change the collaborative relationships between annotators. Among them, when adjusting the collaborative network, the technical means used can still be to make dynamic adjustments based on the calculated Jaccard similarity between different annotators.

[0071] As can be seen from the example above, the collaborative network actually divides the annotators in the data annotation team into multiple collaborative groups, and the annotators in each collaborative group have a high degree of annotation consistency.

[0072] Furthermore, during the dynamic reconstruction of the collaborative network topology, the appropriate annotation task type for each annotator can be dynamically determined based on their historical annotation performance, serving as an attribute information of the annotator. That is, in the collaborative network, a node represents an annotator, and the node information of a node can include the appropriate annotation task type for the represented annotator. Subsequently, in assigning annotation tasks to annotators, appropriate annotation tasks are assigned to different annotators based on the collaborative relationships between them reflected in the collaborative network and the appropriate annotation task types.

[0073] For example, annotator A might be strong in image annotation tasks but weak in text annotation tasks. In this case, annotator A is best suited for image annotation tasks. Suppose an autonomous driving data annotation task involves image annotation, 3D point cloud data annotation, and text annotation. Based on the collaboration network, annotator A has collaborative relationships with annotators B and C. Annotator B is strong in text and audio annotation tasks, while annotator C is strong in point cloud data and video annotation tasks. In this scenario, the autonomous driving data annotation task can be assigned to annotators A, B, and C, who will then collaborate to complete the three annotation stages of the task.

[0074] After the above processing, the team evolution system outputs optimized data annotation team capabilities. This optimized data annotation team capability output is not a single data point, but rather a comprehensive, quantifiable, and structured representation of team capabilities. It reflects the improvements in knowledge, annotation skills, and collaboration efficiency of each annotator in the data annotation team after the "evolution" process, including personalized training programs, dynamic restructuring of the collaboration network, and adjustments to the types of annotation tasks. For example, the output information of the team evolution system may include: each annotator's annotation accuracy, annotation time, annotation effect, decision path information, etc. This output information serves as input to the real-time feedback controller 15, enabling the real-time feedback controller 15 to determine the group consistency index of the data annotation team and trigger corresponding operations based on the group consistency index. The operations triggered by the real-time feedback controller 15 based on the group consistency index will be detailed in the section on "Real-time Feedback Controller 15" below, and will not be elaborated upon here.

[0075] 4) Real-time feedback controller 15 The real-time feedback controller 15 can monitor metrics such as the group consistency index (e.g., the Kappa coefficient). When the group consistency index is found to be below the target (e.g., falling below a preset threshold β), it will trigger the following actions: initiating experience sharing and / or adjusting the collaboration network.

[0076] In the above, initiating experience sharing can be understood as triggering an expert experience broadcasting mechanism. When this mechanism is triggered, the expert's annotation experience templates can be pushed to the annotators. In practice, this doesn't necessarily mean simply sending all expert annotation experience templates to all annotators. Instead, personalized recommendations can be made by analyzing the factors contributing to a decrease in the group consistency index. For example, if "medical terminology" data annotation tasks are found to be a factor causing a decrease in the group consistency index, annotation experience templates related to "medical terminology" data annotation tasks can be pushed to each annotator in the data annotation team. As another example, if a annotator's (e.g., a novice) poor annotation skills are found to be a factor causing a decrease in the group consistency index, suitable annotation experience templates can be pushed to them based on their skill level (e.g., push decision path templates related to boundary annotation tasks).

[0077] The annotation experience templates pushed out are selected from the aforementioned expert experience template library. The forms of pushing annotation experience templates include, but are not limited to, at least one of the following: text and image guides, short videos / animated GIFs, interactive tutorials, and FAQ cards (a type of question-and-answer card). The channels for pushing annotation experience templates may include, but are not limited to: pop-ups, instant messaging channels, email, and training modules provided on data annotation platforms.

[0078] This solution, through the aforementioned "expert experience broadcasting mechanism," can deliver personalized annotation experience and knowledge based on the differences in annotators' abilities. For example, it can provide beginners with expert decision-making path templates for handling ambiguous cases, rather than generalized rule documents. This effectively addresses individual weaknesses, accelerates the integration of new annotators, shortens the team's capability iteration cycle, and avoids the inefficiency of "one-size-fits-all" training.

[0079] Furthermore, the phrase "start adjusting the collaboration network" mentioned above can be understood as sending a notification message to the experience distillation engine to instruct it to adjust the collaboration network. Before adjusting the collaboration network, it also involves re-identifying annotation experts and updating the annotation experience knowledge graph based on the updated annotation behavior logs of each annotator.

[0080] In addition to the above, when the group consistency index is detected to be below the target, the real-time feedback controller 15 can also trigger a feedback operation to send a group consistency assessment report to the administrator. This group consistency assessment report can reflect in which aspects the data annotation team has a higher degree of inconsistency.

[0081] This solution, through the real-time feedback control based on the group consistency index, constructs a closed-loop team evolution system of "annotation expert identification → experience distillation → team optimization → feedback correction". It can achieve proactive optimization of team data annotation quality rather than passive correction, and can form a two-way enhancement between annotation experts and the group (expert experience guides team evolution, and group feedback feeds back to improve expert capabilities).

[0082] Based on the technical architecture described above, this specification also... Figure 2 This illustrates the data annotation team capability evolution process achieved using this technical architecture. For example... Figure 2 As shown, the process of team capability evolution includes the following steps: Step 1: Collect annotation behavior logs of each annotator in the data annotation team. This step, without relying on additional hardware, uses an annotation platform to collect annotation tasks (including the raw data to be annotated), annotation holographic data generated by each annotator when performing the annotation tasks (including annotation process behavior data (such as annotation task completion time, rule application records and other non-biological feature data), and annotation results), and outputs annotation behavior logs for each annotator based on the collected data.

[0083] The annotation behavior logs of each annotator are typically stored in a behavior database for management.

[0084] Step 2: Dynamically identify high-value annotation experts This step involves parsing and analyzing the annotation behavior logs of each annotator stored in the behavior database to extract multidimensional behavioral characteristics of each annotator (such as annotation task completion quality, response timing stability, and rule application consistency). Then, using methods such as dynamic threshold filtering, high-value annotation experts can be selected based on these multidimensional behavioral characteristics, rather than fixed selection criteria. Furthermore, an expert feature fingerprint database is output. This database stores the multidimensional behavioral characteristics of each annotation expert, as well as annotation process behavior data.

[0085] Step 3: Extracting Expert Decision-Making Models This step utilizes a pre-built decision tree model to perform in-depth analysis of the annotation process behavior data of each annotation expert stored in the expert feature fingerprint database. This transforms the various annotation behaviors of the annotation experts into quantifiable annotation knowledge, such as annotation paths. Furthermore, generative adversarial networks can be used to generate adversarial examples to simulate complex data annotation tasks (such as boundary labeling tasks) that annotation experts excel at handling. Further, an expert experience template library is output, containing annotation experience templates from each annotation expert.

[0086] Step 4: Construct a dynamically labeled experience knowledge graph This step involves structuring the annotation experience templates (such as annotation rule application templates, annotation task case templates, and error correction logic templates) stored in the expert experience template library, incrementally updating the annotation experience knowledge graph, and outputting the incrementally updated annotation experience knowledge graph. This annotation experience knowledge graph contains entities (such as annotation rules and annotation task cases) and relationships between entities (such as correction -> avoidance -> etc.).

[0087] Step 5: Optimize the collaborative network of the data annotation team. This step dynamically adjusts the collaborative relationships among annotators based on the annotation experience knowledge graph and the current team status of the data annotation team (e.g., annotators with high similarity form collaborative groups), and dynamically allocates annotation tasks based on differences in annotators' abilities and / or the collaborative relationships between annotators (e.g., complex annotation tasks are preferentially assigned to annotation experts). Furthermore, it outputs an optimized collaborative network.

[0088] Step 6: Monitor the group consistency of the data annotation team in real time. This step calculates a group consistency index (such as the Kappa coefficient) based on the annotation results, accuracy, and time taken by each annotator in the data annotation team. When the group consistency index drops below the acceptable level, a feedback mechanism is triggered to output a consistency assessment report to the administrator.

[0089] Additionally, when group consistency is not met, expert experience broadcasting will be triggered (such as pushing the annotation experience template of the annotation expert to all annotators), and the process will return to step 2 to trigger the adjustment of the collaboration network.

[0090] Furthermore, it will output an updated collaboration strategy (i.e., an adjusted collaboration network).

[0091] Sterp7, optimized data annotation team The goal of this step is to improve annotation quality and population consistency, and to adapt to the needs of large model iteration.

[0092] The technical architecture mentioned above can be implemented based on a server-side and client-side architecture. See further details... Figure 1 As shown, the data acquisition module 11 in the technical architecture can be deployed on the client side, while the expert feature modeling module 12, experience distillation engine 13, team evolution system 14, real-time feedback controller 15, etc., can be deployed on the server side. The server side can be a server, server cluster, virtual server, or cloud, etc. The client side can be, but is not limited to, smartphones, smart wearable devices, tablets, laptops, desktop computers, etc.

[0093] thus, Figure 3 This specification also illustrates a team capability optimization system according to an embodiment, which includes a client 200 and a server 100. Client 200 is used to display the client interface and collect the behavior operations performed by each member of the team through the client interface while performing work tasks, and send the behavior operations to server 100. Server 100 is configured to record the received behavioral operations in the corresponding task execution behavior log of the member; and to further implement: obtaining the task execution behavior characteristics of each member in the team; the task execution behavior characteristics are extracted by analyzing the task execution behavior log of the member and can reflect the member's work task execution ability; calculating the ability score of each member based on the task execution behavior characteristics; determining a score threshold based on the ability score of each member; performing a member screening operation based on the score threshold and the ability score of each member to determine the screened members as task experts; determining the relevant task behavior information of the task experts based on the task execution behavior characteristics and the task execution behavior log; and performing a team capability optimization operation based on the relevant task behavior information of the task experts and the current team capability status of the team; wherein the team capability optimization operation includes at least one of the following: pushing task execution training plans to members in the team and adjusting the task collaboration relationship between members in the team.

[0094] In the above context, the term "team" can refer to, but is not limited to, a data annotation team, a software development team, etc. In the context of a data annotation team, the "members" are the annotators, and the "tasks" are the data annotation tasks.

[0095] The specific implementation of the functions of server 100 and client 200 will be described in detail in the following method embodiments, and will not be elaborated here.

[0096] The technical solutions provided in this specification will be described below by way of method embodiments.

[0097] Figure 4 This diagram illustrates a flowchart of a team capability optimization method according to an embodiment of this specification. The method is executed by the server 100 in the aforementioned system. See also... Figure 4 As shown, the team capability optimization method includes the following steps: 102. Obtain the task execution behavior characteristics of each member in the team; the task execution behavior characteristics are extracted by analyzing the task execution behavior logs of the members and can reflect the members' work task execution capabilities; 104. Based on the aforementioned task execution behavior characteristics, calculate the ability score for each member; 106. Determine the score threshold based on each member's ability score; 108. Based on the scoring threshold and the ability scores of each member, perform a member screening operation to identify the screened members as task experts; 110. Based on the task execution behavior characteristics and the task execution behavior log of the task expert, determine the relevant task behavior information of the task expert; 112. Based on the relevant task behavior information of the task expert and the current team capability status of the team, optimize the team's capabilities; wherein, the capability optimization includes at least one of the following: pushing task execution training programs to members to compensate for the team's shortcomings in task execution capabilities, and adjusting the task collaboration relationships among members.

[0098] In different scenarios, the terms "team", "member", "work task" and "task execution log" used in this embodiment refer to different things.

[0099] For example, in a data annotation scenario, the term "team" refers to a data annotation team. Correspondingly, team members are annotators, work tasks are data annotation tasks, and task execution logs are annotation behavior logs. Specifically, annotators include initial annotators and reviewers. Initial annotators are those who perform the initial annotation for data annotation tasks, primarily responsible for labeling the raw data according to annotation rules. Their annotation results often require review by a reviewer. Reviewers are those responsible for reviewing and verifying the annotation results; reviewers often have the authority to correct the annotation results.

[0100] Of course, in some other embodiments, "team" can also refer to other types of teams, such as software development teams. This embodiment does not specifically limit the type of "team".

[0101] In the detailed description of the specific implementation of each step provided in this embodiment, we mainly use "team" as the data labeling team and "work task" as the data labeling task as examples.

[0102] In step 102 above, the member's task execution behavior log is generated by recording various behavioral operations performed during the execution of work tasks. By analyzing the member's task execution behavior log, multi-dimensional task execution behavior features that characterize the member's task execution ability can be extracted. For example, multi-dimensional task execution behavior features may include: the quality of work task completion (which can be determined by analyzing the accuracy of the annotation results), the stability of the work task response sequence (which can be determined by analyzing the complete time consumption of the work task), and so on. That is, it can be understood that the task execution behavior features are derived and analyzed based on the information recorded in the behavior log, and belong to derived behavior features.

[0103] For example, in a data annotation scenario, the task execution behavior log is an annotation behavior log, which records the annotator's annotation process behavior data (such as the total time spent on the data annotation task, annotation rule application records, error correction paths, etc.) and annotation results. Multi-dimensional annotation behavior characteristics of the annotator can be extracted from the annotation behavior log. These multi-dimensional annotation behavior characteristics include, but are not limited to: data annotation task completion quality, response time stability, consistency of annotation rule application, cross-task transferability, and knowledge dissemination efficiency.

[0104] For a detailed description of the annotation behavior log, annotation process behavior data, annotation behavior features, etc. in this example, please refer to the aforementioned combination Figure 1 Regarding the relevant content described in "Data Acquisition Module 11".

[0105] In steps 104, 106, and 108 above, the multi-dimensional task execution behavior characteristics of each member can be weighted and fused to calculate the ability score (also known as the comprehensive ability score) of each member. Further, statistical analysis is performed on the ability scores of each member to determine the corresponding mean score and score-label difference, and then a score threshold is dynamically determined based on these mean score and score-label difference. Afterwards, members can be sorted according to their ability scores from largest to smallest, and the top N members with ability scores greater than or equal to the score threshold are identified as task experts. Here, N is a positive integer, such as 1, 2, or 3, which determines at least one task expert. In the data labeling scenario, the "task expert" here is specifically called a "labeling expert."

[0106] For the specific implementation of steps 104, 106, and 108, please refer to the aforementioned combination Figure 1 The relevant content is given when introducing "Expert Feature Modeling Module 12".

[0107] In steps 110 and 112 above, the relevant task behavior information of the task expert includes: task execution behavior characteristics (such as labeled behavior characteristics) obtained through step 102, and task execution process behavior data (such as labeled process behavior data). The task execution process behavior data can be obtained from the task execution behavior log. Of course, in some other embodiments, the relevant task behavior information may also include other information contained in the task execution behavior log, such as task execution results and work task attributes (such as task execution difficulty). This information can then be used to determine the complex tasks that the task expert is good at handling. Furthermore, the current team capability status is a group-level cognitive indicator, representing the overall behavioral characteristics of the team during task execution. For a detailed description of the current team capability status, please refer to the foregoing combination... Figure 1 The introduction to "Team Evolution System 14" includes content related to "data labeling the current team capability status of the team".

[0108] Based on the task expert's relevant task behavior information and the current team capability status, the experience knowledge graph can be updated, the team's task execution capability weaknesses can be identified, and personalized task execution training programs can be generated to address these weaknesses. Therefore, in one feasible implementation, step 112, "optimizing the team's capabilities based on the task expert's relevant task behavior information and the team's current team capability status," may include: 1122. Based on the relevant task behavior information of at least one of the task experts, incrementally update the experience knowledge graph; 1124. Based on the current team capability status, identify the team's weaknesses in task execution capabilities; 1126. Based on the relevant task behavior information of at least one of the task experts, determine a target task expert from the at least one task expert who can compensate for the shortcomings in task execution capability; 1128. Based on the experiential knowledge in the experiential knowledge graph that is related to the expert of the target task and can compensate for the shortcomings of the task execution capability, generate a task execution training program; 11210. Push the task execution training plan to each member of the team.

[0109] In step 1122 above, a task execution experience template of a task expert can be generated first, and then the experience knowledge graph can be incrementally updated based on the task execution experience template. That is, a specific implementation scheme of step 10122 above may include: S22. Based on the relevant task behavior information of the task expert, generate the task execution experience template of the task expert; S24. Store the task execution experience template into the experience template library; S26. Based on the task execution experience templates stored in the experience template library, incrementally update the experience knowledge graph.

[0110] In the above context, the "task execution experience template" can be specifically referred to as the "annotation experience template" and the "experience knowledge graph" can be specifically referred to as the "annotation experience knowledge graph".

[0111] For a detailed explanation of the specific implementation of steps S22, S24, and S26 above, as well as the task execution experience template and experience knowledge graph, please refer to the aforementioned combination Figure 1 The content related to "building annotated experience knowledge graphs" given when introducing Experience Distillation Engine 13 will not be elaborated on here.

[0112] In steps 1124, 1126, 1128, and 11210 above, based on the current team capability status, weak links in the team's task execution can be identified. These weak links constitute the team's shortcomings in task execution capability. Further, suitable target task experts can be selected. Then, using the experience knowledge graph, relevant experience knowledge from the target task expert that can compensate for the shortcomings in task execution capability, a task execution training plan can be generated and pushed to all team members for learning. The task execution training plan can take the form of micro-lecture videos, and the channels for distribution can be, but are not limited to, email, training modules provided by data annotation platforms, pop-ups, instant messaging channels, etc.

[0113] In the context of data annotation, the "shortcomings in task execution capability" mentioned here can be specifically referred to as "shortcomings in annotation capability," and the "task execution training program" can be specifically referred to as "annotation training program."

[0114] For specific implementations of 1124, 1126, 1128, and 11210 mentioned above, please refer to the aforementioned combination. Figure 1 The content related to "generating personalized training programs" is given when introducing Team Evolution System 14.

[0115] Furthermore, the solution provided in this embodiment can also simulate complex tasks that task experts are adept at handling to generate adversarial tasks. These adversarial tasks can be used to train teams, thereby improving the team's ability to handle complex tasks. That is, the method provided in this embodiment may further include the following steps: S32. Based on the relevant task behavior information of the task expert, determine the complex tasks that the task expert is good at performing, including work tasks with ambiguity in task execution. S34. Simulate the complex task to generate adversarial tasks; S36. The adversarial task is pushed to each member of the team for task execution practice.

[0116] As described above, the complexity of the tasks performed by the task experts and the accuracy of the results can be used to determine which complex tasks they are good at, such as tasks with ambiguity (e.g., fuzzy labeling). Further, the complex task is input into a Generative Adversarial Network (GAN). The GAN can generate adversarial tasks by adding perturbations to the complex task and then output the adversarial task. Afterwards, the adversarial task can be pushed to team members for practice through appropriate channels, such as email, pop-ups, training modules provided by the task platform, and instant messaging.

[0117] For specific implementations of S32, S34, and S36, please refer to the aforementioned combination Figure 1 The content related to "adversarial tasks" given in the introduction to "Constructing Annotated Experience Knowledge Graphs" is as follows.

[0118] Furthermore, this embodiment can also realize the construction and dynamic adjustment of the team's collaborative network to optimize the collaborative relationships among team members and achieve reasonable classification of work tasks. Therefore, the method provided in this embodiment may also include the following steps: S42. Based on the task execution behavior logs of each member, determine the task types that each member is suitable for executing, and calculate the similarity of task execution performance between each pair of members; S44. Determine the task collaboration relationship between members based on the similarity of task execution performance; S46. Adjust the team's collaboration network based on the task types performed by the members and the task collaboration relationships between the members; The collaborative network is used to guide the allocation of work tasks; it includes multiple nodes and relationships between nodes; a node represents a member of the team, and the node information of a node includes the member's member identifier and the type of task the member is suited to perform; the relationships between nodes reflect the task collaboration relationships among members. As described above, the appropriate task type for each member can be determined based on their task execution results, execution time, and behavioral data. For example, Jaccard similarity or cosine distance can be used to calculate the similarity of task performance between pairs of members. When the similarity of task performance between two members exceeds a set similarity threshold, it can be determined that these two members have similar task performance, can collaborate on the same type of task, or collaborate on different stages of the same task. Furthermore, based on the appropriate task type for each member, the node information representing that member in the collaboration network can be updated. And based on the collaborative relationships between members, "edges" (connections between two nodes) in the collaboration network can be established or updated. This collaboration network can then guide the allocation of tasks to each member.

[0119] For specific implementations of S342, S44, and S46, please refer to the foregoing combination Figure 1 The content related to "dynamically reconstructing the topology of collaborative networks" is given when introducing "Team Evolution System 14".

[0120] Furthermore, after optimizing the team's capabilities by pushing task execution training plans to the team, adjusting the team's collaboration network to optimize the collaborative relationships among members, and updating the task types that members can perform, in order to evaluate the effectiveness of capability optimization based on the real-time monitored consistency of the team's task execution (such as the group consistency index), and if it is determined that the capability optimization effect is not up to standard, the team capability optimization can be resumed. For example, it can push suitable task execution experience templates to members and / or readjust the collaboration network.

[0121] Therefore, the solution provided in this embodiment may also include the following steps: 114. Calculate the team's task execution consistency based on the task execution behavior logs generated by each member after the capability optimization is implemented; 116. When the consistency of task execution is less than or equal to a preset threshold, a suitable task execution experience template is pushed to team members. Specifically, the influencing factors that cause the consistency of task execution to be less than the preset threshold can be analyzed first; then, based on the influencing factors, a suitable task execution experience template is selected from the experience template library; and the selected task execution experience template is pushed to the corresponding members of the team.

[0122] The degree of consistency in task execution mentioned above can be evaluated comprehensively based on one or more of the following dimensions, but is not limited to: consistency of task execution results (such as annotation results) among members for the same batch of work tasks (as a group consistency index, such as the Kappa coefficient), similarity of task execution time (such as annotation time), and pattern matching of behavioral decision-making paths.

[0123] When the consistency of task execution falls below a preset threshold, an expert experience broadcasting mechanism is triggered to push suitable task execution experience templates (such as annotated experience templates) to relevant team members. For details on the specific implementation of "pushing suitable task execution experience templates to relevant team members," please refer to the other embodiments mentioned above. Figure 1 The article introduces the annotation experience template and related content provided when introducing the "Real-time Feedback Controller 15".

[0124] Furthermore, the method provided in this embodiment may also include the following steps: 118. When the team capability optimization trigger condition is met, the following actions are triggered: Analyze the task execution behavior logs generated by each member after the capability optimization is performed, and extract the updated task execution behavior characteristics of each member; re-select task experts based on the updated task execution behavior characteristics of each member; and perform capability optimization on the team again based on the relevant task behavior information of the re-selected task experts and the current team capability status of the team.

[0125] The conditions for meeting the team capability optimization trigger include at least one of the following: the consistency of task execution is less than a preset threshold, or the set team capability optimization cycle is reached. The team capability optimization cycle includes: a time interval cycle (e.g., once every 24 hours, once every Monday at 9:00 AM) and / or an event cycle based on the cumulative number of task executions (e.g., once after every 100 complete data labeling tasks).

[0126] For details regarding the extraction of updated task execution behavior characteristics of members, re-screening of task experts, and optimization of the team's execution capabilities, please refer to the relevant content in steps 102, 104, 106, 108, 110, and 112 mentioned above.

[0127] The above text combined Figure 4Specific embodiments of the embodiments described herein have been described. It should be noted that other embodiments are within the scope of the appended claims. Furthermore, in some cases, the actions or steps described in the specification may be performed in a different order than those shown in the embodiments and still achieve the desired results. Additionally, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0128] The apparatus embodiments corresponding to the method embodiments provided in this specification are described below.

[0129] Figure 5 A schematic diagram of a team capability optimization device provided in an exemplary embodiment of this specification is shown. Figure 5 As shown, the device includes: an acquisition module 52, a calculation / determination module 54, an execution module 56, and an optimization module 58. Among them, The acquisition module 52 is used to acquire the task execution behavior characteristics of each member in the team; the task execution behavior characteristics are extracted by analyzing the task execution behavior logs of the members and can reflect the members' work task execution capabilities. The calculation / determination module 54 is used to calculate the ability score of each member based on the task execution behavior characteristics; and determine the score threshold based on the ability score of each member; Execution module 56 is used to perform a member screening operation based on the scoring threshold and the ability scores of each member to determine the screened members as task experts; The calculation / determination module 54 is further configured to determine the relevant task behavior information of the task expert based on the task execution behavior characteristics and the task execution behavior log of the task expert; The optimization module 58 is used to optimize the capabilities of the team based on the relevant task behavior information of the task expert and the current team capability status of the team; wherein, the capability optimization includes at least one of the following: pushing task execution training programs to members to compensate for the team's task execution capability shortcomings, and adjusting the task collaboration relationship between members.

[0130] In one possible implementation, the number of task experts is at least one. Furthermore, the optimization module 58, when optimizing the team's capabilities based on the relevant task behavior information of the task experts and the team's current capability status, specifically performs the following: incrementally updating the experience knowledge graph based on the relevant task behavior information of at least one task expert; identifying the team's task execution capability shortcomings based on the current team capability status; determining a target task expert from the at least one task expert who can compensate for the task execution capability shortcomings based on the relevant task behavior information of at least one task expert; generating a task execution training plan based on the experience knowledge in the experience knowledge graph that is related to the target task expert and can compensate for the task execution capability shortcomings; and pushing the task execution training plan to each member of the team.

[0131] The aforementioned optimization module 58, when used to incrementally update the experience knowledge graph based on the relevant task behavior information of the at least one task expert, is specifically used to: generate a task execution experience template for the task expert based on the relevant task behavior information of the task expert; store the task execution experience template in an experience template library; and incrementally update the experience knowledge graph according to the task execution experience template stored in the experience template library.

[0132] In one possible implementation, the aforementioned calculation / determination module 54 is further configured to determine, based on relevant task behavior information of the task expert, the complex tasks that the task expert is adept at performing, including work tasks with ambiguity in task execution. The device also includes a generation module and a push module. The generation module is configured to simulate the complex tasks to generate adversarial tasks. The push module is configured to push the adversarial tasks to each member of the team for task execution practice.

[0133] In one possible implementation, the calculation / determination module 54 is further configured to determine the task types suitable for each member to execute based on the task execution behavior logs of each member, and calculate the task execution performance similarity between pairs of members; and determine the task collaboration relationship between members based on the task execution performance similarity. The device also includes an adjustment module, configured to adjust the team's collaboration network based on the task types suitable for each member to execute and the task collaboration relationship between members. The collaboration network is used to guide the allocation of work tasks; the collaboration network includes multiple nodes and relationships between nodes; a node represents a member of the team, and the node information of a node includes the member's member identifier and the task type suitable for that member to execute; the relationships between nodes reflect the task collaboration relationship between members.

[0134] In one possible implementation, the calculation / determination module 54 calculates the task execution consistency level of the team based on the task execution behavior logs generated by each member after the capability optimization. The execution module 56 is further configured to, when the task execution consistency level is less than or equal to a preset threshold, perform the following steps: analyze the influencing factors that cause the task execution consistency level to be less than the preset threshold; select a suitable task execution experience template from the experience template library based on the influencing factors; and push the selected task execution experience template to the corresponding members of the team.

[0135] In one possible implementation, the device further includes a triggering module, configured to: upon determining that the team capability optimization triggering condition is met, trigger the following actions: analyzing the task execution behavior logs generated by each member after the capability optimization operation is performed, and extracting the updated task execution behavior characteristics of each member; re-selecting task experts based on the updated task execution behavior characteristics of each member; and performing capability optimization on the team again based on the relevant task behavior information of the re-selected task experts and the current team capability status of the team. The team capability optimization triggering condition includes at least one of the following: the task execution consistency level is less than or equal to a preset threshold, or a set team capability optimization cycle is reached, wherein the team capability optimization cycle includes: a time interval cycle and / or an event cycle based on the cumulative number of task executions.

[0136] In one possible implementation, the team is a data annotation team, the members of the team are annotators, and the work task is a data annotation task. Furthermore, the task execution behavior characteristics of the members include at least one of the following: data annotation task completion quality, consistency of annotation rule application, and response time stability; the response time stability reflects the degree of fluctuation in the time consumed by members in completing similar data annotation tasks.

[0137] It should be noted that the above-mentioned devices can implement the technical solutions described in the corresponding method embodiments. The specific implementation principles of each module or unit can be found in the relevant content of the corresponding method embodiments, and will not be elaborated further here. Furthermore, for ease of description, the above devices are described by function as various modules or units. Of course, when implementing one or more of this specification, the functions of each module or unit can be implemented in one or more software and / or hardware, or a module that implements the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0138] Furthermore, embodiments of this specification also provide an electronic device. For example... Figure 6 As shown, the electronic device 900 includes a memory 91 and a processor 92.

[0139] The aforementioned memory 91 can be implemented by at least one volatile or non-volatile storage device of any type, or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. Furthermore, the memory, wholly or partially, can be integrated with the processor. The memory can contain both removable and non-removable components.

[0140] The processor 92 described above may include one or more general-purpose processors and / or special-purpose processors.

[0141] Furthermore, memory 91 may contain a non-transitory computer-readable medium storing executable program instructions 912 (e.g., compiled or uncompiled program logic and / or machine code). Processor 92 is capable of executing the program instructions 912 stored in memory to implement any methods, processes, or functions disclosed in this specification and / or the accompanying drawings. Additionally, execution of program instructions 912 by processor 92 may result in processor using corresponding data 911.

[0142] For example, the program instructions 912 described above may include an operating system 9122 (e.g., an operating system kernel, device drivers, and / or other modules) installed on the electronic device 900, and one or more application programs 9121 (e.g., a browser, social media application, or game application). Similarly, the data 911 described above may include operating system data 9112 and application data 9111. The operating system data 9112 is primarily accessible to the operating system 9122, while the application data 9111 is primarily accessible to one or more application programs 9121. The application data 9111 may reside in a file system visible or hidden from the user of the electronic device 900.

[0143] Application 9121 can communicate with operating system 9122 through one or more application programming interfaces (APIs). These APIs facilitate application 9121 in reading and / or writing application data, transmitting or receiving information via communication components, and receiving or displaying information on the user interface. In some terms, application 9121 may be simply referred to as "app". Furthermore, application 9121 can be downloaded to the electronic device through one or more online application stores or app markets. However, application 9121 can also be installed on electronic device 400 in other ways, such as through a web browser or a physical interface on electronic device 900 (e.g., a USB port). Furthermore, such as Figure 6 As shown, the electronic device also includes other components such as a communication component 93, a display 94, a power supply component 95, an audio component 96, and a user interface 97. Figure 6 The diagram only shows some components and does not mean that the electronic device 900 includes only these components. Figure 6 The components shown. Additionally... Figure 6 The components within the dashed box are optional, not mandatory, and their specific requirements depend on the product form of the electronic device 900. The electronic device 900 in this embodiment can be a terminal device such as a desktop computer, laptop computer, smartphone, or IoT device; it can also be a server-side device such as a conventional server, cloud server, or server array; or it can be an integrated device combining terminal and server-side devices. If the electronic device 900 in this embodiment is implemented as a terminal device such as a desktop computer, laptop computer, or smartphone, it may include... Figure 6 The components within the dashed box; if the electronic device 900 in this embodiment is implemented as a server-side device such as a conventional server, cloud server, or server array, then it may not include... Figure 6 The component within the dashed box.

[0144] The aforementioned communication component 93 is configured to facilitate wired or wireless communication between the device housing the communication component and other devices. The device housing the communication component 93 can access wireless networks based on communication standards, such as 2G, 3G, 4G / LTE, 5G, or combinations thereof. In one exemplary embodiment, the communication component 93 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. Specifically, the communication component 93 includes a communication interface that enables the electronic device 900 to communicate with other devices, access networks, and transmission networks via analog or digital modulation. For example, the communication interface may include a chipset and antenna for wireless communication with a radio access network or access point. Furthermore, the communication interface can be a wired interface such as Ethernet, Token Ring, or a USB port, or a wireless interface such as Wi-Fi, Bluetooth, Global Positioning System (GPS), or a wide-area wireless interface (e.g., WiMAX or LTE). Of course, the communication interface can also support other forms of physical layer interfaces and standard or proprietary communication protocols. The communication interface may also include multiple physical communication interfaces, such as Wi-Fi, Bluetooth, and wide-area wireless interfaces.

[0145] The aforementioned display 94 includes a screen, which may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen can be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation.

[0146] The power supply component 95 provides power to various components of the device in which it resides. The power supply component 95 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply component resides.

[0147] The aforementioned audio component 96 can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals can be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0148] The user interface 97 described above includes receiving user input and providing output to the user. Therefore, the user interface 97 may include input components such as a keypad, keyboard, touch-sensitive or presence-sensitive panel, computer mouse, trackball, joystick, microphone, still camera, and video camera, and output components such as a display screen (which may be combined with a touch-sensitive panel), CRT, LCD, LED, display using DLP technology, printer, and other similar devices known or developed in the future. The user interface 97 may also generate auditory output via speakers, speaker jacks, audio output ports, audio output devices, headphones, and other similar devices known or developed in the future. In some embodiments, the user interface 97 may include software, circuitry, or other forms of logic capable of transmitting and receiving data from external user input / output devices. Additionally or alternatively, the electronic device 900 may support remote access from other devices via a communication interface or another physical interface (not shown). The user interface 97 may be configured to receive user input, the position and movement of which may be indicated by indicators or cursors described herein. The user interface 97 may also be configured as a display device for rendering or displaying text fragments.

[0149] Accordingly, embodiments of this specification also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the above-described method embodiments. The computer-readable storage medium includes volatile or non-volatile or a combination thereof, and can be removable or non-removable. Examples of computer-readable storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), flash memory or other memory technologies, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium. Furthermore, embodiments of this specification also provide a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed in a computer, it causes the computer to perform actions such as... Figure 4 The method described.

[0150] This specification also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implements... Figure 4 The method described.

[0151] Those skilled in the art will recognize that the functions described in the various embodiments disclosed in this specification in one or more of the examples above can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0152] The specific embodiments described above further illustrate the purpose, technical solutions, and beneficial effects of the multiple embodiments disclosed in this specification. It should be understood that the above descriptions are merely specific implementations of the multiple embodiments disclosed in this specification and are not intended to limit the protection scope of the multiple embodiments disclosed in this specification. Any modifications, equivalent substitutions, improvements, etc., made based on the technical solutions of the multiple embodiments disclosed in this specification should be included within the protection scope of the multiple embodiments disclosed in this specification.

Claims

1. A team capability optimization method characterized by, include: Obtain the task execution behavior characteristics of each team member; The task execution behavior characteristics are extracted by analyzing the task execution behavior logs of the members and can reflect the members' work task execution capabilities. Based on the aforementioned task execution behavior characteristics, calculate the ability score of each member; Determine the score threshold based on each member's ability score; Based on the scoring threshold and the ability scores of each member, a member screening operation is performed to identify the screened members as task experts; Based on the task execution behavior characteristics and task execution behavior log of the task expert, the relevant task behavior information of the task expert is determined; Based on the relevant task behavior information of the task experts and the current team capability status, the team's capabilities are optimized; wherein, the capability optimization includes at least one of the following: pushing task execution training programs to members to compensate for the team's shortcomings in task execution capabilities, and adjusting the task collaboration relationships among members.

2. The method of claim 1, wherein, The number of task experts is at least one; Furthermore, based on the relevant task behavior information of the task experts and the current team capability status, the team's capabilities are optimized, including: Based on the relevant task behavior information of at least one of the task experts, the experience knowledge graph is incrementally updated. Based on the current team capability status, identify the team's weaknesses in task execution capabilities; Based on the relevant task behavior information of at least one of the task experts, a target task expert that can compensate for the shortcomings in task execution capability is determined from at least one of the task experts. Based on the experiential knowledge in the experiential knowledge graph that is relevant to the experts of the target task and can compensate for the shortcomings in the task execution capability, a task execution training program is generated. The task execution training plan is then pushed to each member of the team.

3. The method according to claim 2, characterized in that, Based on the relevant task behavior information of at least one of the task experts, the experience knowledge graph is incrementally updated, including: Based on the relevant task behavior information of the task expert, a task execution experience template of the task expert is generated; The task execution experience templates are stored in the experience template library; The experience knowledge graph is incrementally updated based on the task execution experience templates stored in the experience template library.

4. The method according to claim 2 or 3, characterized in that, Also includes: Based on the relevant task behavior information of the task expert, the complex tasks that the task expert is good at performing are determined. The complex tasks include work tasks with ambiguity in task execution. The complex task is simulated to generate adversarial tasks; The adversarial task is then pushed to each member of the team for task execution practice.

5. The method according to claim 2 or 3, characterized in that, Also includes: Based on the task execution behavior logs of each member, determine the types of tasks that each member is suitable to execute, and calculate the similarity of task execution performance between each pair of members; Based on the similarity of task performance, the task collaboration relationships between members are determined; Based on the task types that the members are suited to perform and the task collaboration relationships between the members, the team's collaboration network is adjusted; The collaboration network is used to guide the allocation of work tasks; the collaboration network includes multiple nodes and relationships between nodes; a node represents a member of the team, and the node information of a node includes the member's member identifier and the type of task that the member is adapted to perform; The relationships between nodes reflect the collaborative relationships among members.

6. The method according to any one of claims 2 to 4, characterized in that, Also includes: The task execution consistency of the team is calculated based on the task execution behavior logs generated by each member after the capability optimization is performed. When the consistency of task execution is less than or equal to a preset threshold, then: analyze the influencing factors that cause the consistency of task execution to be less than the preset threshold; Based on the aforementioned influencing factors, a suitable task execution experience template is selected from the experience template library; the selected task execution experience template is then pushed to the corresponding members of the team.

7. The method according to claim 6, characterized in that, Also includes: When the team capability optimization trigger condition is met, the following is triggered: Analyze the task execution behavior logs generated by each member after the capability optimization operation is performed, and extract the updated task execution behavior characteristics of each member. Based on the updated task execution behavior characteristics of each member, task experts were re-selected. Based on the relevant task behavior information of the re-selected task experts and the current team capability status, the team performs capability optimization again. The conditions for meeting the team capability optimization trigger include at least one of the following: the consistency of task execution is less than or equal to a preset threshold, or the set team capability optimization cycle is reached. The team capability optimization cycle includes a time interval cycle and / or an event cycle based on the cumulative number of task executions.

8. The method according to any one of claims 2 to 4, characterized in that, The team is a data annotation team, the members of the team are annotators, and the work task is a data annotation task. Furthermore, the task execution behavior characteristics of the members include at least one of the following: data annotation task completion quality, consistency of annotation rule application, and response timing stability; the response timing stability reflects the degree of fluctuation in the time consumption of members in completing similar data annotation tasks.

9. A team capability optimization system, characterized in that, include: The client is used to display the client interface and collect the actions of each member of the team in performing work tasks through the client interface, and send the actions to the server. The server is configured to record the received behavioral operations in the corresponding member's task execution behavior log; and is also configured to implement the method of any one of claims 1 to 8.

10. A team capability optimization device, characterized in that, include: The acquisition module is used to acquire the task execution behavior characteristics of each member in the team; The task execution behavior characteristics are extracted by analyzing the task execution behavior logs of the members and can reflect the members' work task execution capabilities. The calculation and determination module is used to calculate the ability score of each member based on the task execution behavior characteristics; And based on each member's ability score, a score threshold is determined; The execution module is used to perform a member screening operation based on the scoring threshold and the ability scores of each member to determine the screened members as task experts. The calculation and determination module is further configured to determine the relevant task behavior information of the task expert based on the task execution behavior characteristics and the task execution behavior log of the task expert; An optimization module is used to optimize the capabilities of the team based on the relevant task behavior information of the task expert and the current team capability status of the team; wherein, the capability optimization includes at least one of the following: pushing task execution training programs to members to compensate for the team's shortcomings in task execution capabilities, and adjusting the task collaboration relationships among members.

11. An electronic device, characterized in that, The method includes a memory and a processor, wherein the memory stores executable program instructions, and the processor executes the program instructions to implement the method of any one of claims 1 to 8.

12. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed in a computer, causes the computer to perform the method described in any one of claims 1 to 8.

13. A computer program product, characterized in that, The computer program product includes a computer program or instructions that, when executed by a processor, implement the method of any one of claims 1 to 8.

Images