Labeling task execution method and device, storage medium, equipment and program product

Abstract

The invention discloses a labeling task execution method and device, a storage medium, equipment and a program product. The annotation task execution method comprises the steps that task requirements and characteristics of an annotation task can be effectively described according to a determined task portrait, so that an artificial intelligence model can automatically determine a main body suitable for annotating to-be-annotated data through the task portrait, human intervention is greatly reduced in the whole process, consumption of manpower cost is reduced, and the annotation task execution efficiency is improved. Moreover, it can be ensured that the to-be-labeled data and the main body are highly adaptive, so that the distribution efficiency is greatly improved in the task distribution stage, and the labeling efficiency in the data labeling stage is further improved.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of computer technology, and in particular, to methods, devices, storage media, equipment, and program products for executing annotation tasks. Background Art

[0002] With the continuous development of computer technology, artificial intelligence technology has played a significant role in many business scenarios. It has not only brought great convenience to people's daily lives, but also made great contributions to ensuring the security of users' personal information and privacy data.

[0003] In the specific implementation process, it is usually necessary to deploy corresponding artificial intelligence models in various business scenarios. In order to ensure the performance of artificial intelligence models, early model training is indispensable. By training these models, the models can learn a lot of "knowledge" so as to output accurate results for users in subsequent practical applications.

[0004] Before the model training task begins, a large number of training samples need to be prepared. However, these training samples usually need to complete the sample analysis process through human experience to determine the labelers who are suitable for labeling these samples, and then assign these samples to each labeler for labeling. This analysis process and sample allocation process not only consumes a lot of labor costs, but also there is a situation where the samples and labelers are not compatible (that is, the determined labelers are not suitable for labeling the assigned samples), which leads to low overall labeling efficiency. Summary of the invention

[0005] In view of this, one or more embodiments of this specification provide the following technical solutions: According to a first aspect of one or more embodiments of this specification, a method for executing a labeling task is proposed. The method is applied to a labeling system, in which an artificial intelligence model is deployed, and includes: Get the user's annotation request; Determining the user's data to be annotated and annotation requirement information according to the annotation request, wherein the annotation requirement information is used to indicate the user's annotation requirement for the data to be annotated; Inputting the labeling requirement information into the deployed artificial intelligence model, so as to determine, through the artificial intelligence model, a task profile for describing the labeling task corresponding to the labeling request, and determining at least one subject for performing the labeling task from among the candidate subjects according to the task profile and the subject attribute information of each candidate subject obtained; The data to be labeled is distributed to a terminal device used by the at least one subject to perform the labeling task.

[0006] Optionally, determining at least one subject for performing the labeling task from among the candidate subjects according to the task portrait and the acquired subject attribute information of each candidate subject specifically includes: Determining a task difficulty level corresponding to the labeling task according to the task profile; At least one subject for performing the labeling task is determined from among the candidate subjects according to the task difficulty level and the acquired subject attribute information of each candidate subject.

[0007] Optionally, the subject attribute information of each candidate subject is obtained, including: Based on the task portrait, determining a historical annotation task that matches the annotation task from various historical annotation tasks as a reference annotation task; The subject that has historically performed the reference annotation task is used as a candidate subject, and subject attribute information of the candidate subject is obtained.

[0008] Optionally, the subject attribute information of each candidate subject is obtained, including: The calling instruction generated by the artificial intelligence model calls the preset subject management tool to obtain the subject attribute information of each candidate subject through the called subject management tool. The calling instruction is generated by the artificial intelligence model after receiving the labeling requirement information.

[0009] Optionally, before allocating the data to be labeled to a terminal device used by the at least one subject, the method further includes: Based on the task portrait, determining a labeling template and a labeling auxiliary tool corresponding to the labeling task through the artificial intelligence model; Allocating the data to be annotated to a terminal device used by the at least one subject to perform the annotation task specifically includes: The data to be annotated is distributed to the at least one terminal device used, so that the at least one subject performs the annotation task based on the annotation template and the annotation auxiliary tool.

[0010] Optionally, the annotation auxiliary tool includes: an online auxiliary tool; Allocating the data to be annotated to the at least one terminal device used so that the at least one subject performs the annotation task based on the annotation template and the annotation auxiliary tool specifically includes: The loading link of the annotation template and the online auxiliary tool is sent to the terminal device used by the at least one subject, so that the terminal device obtains the annotation template, and loads the online auxiliary tool based on the loading link, and the at least one subject performs the annotation task based on the annotation template and the online auxiliary tool.

[0011] Optionally, the annotation auxiliary tool includes: an offline auxiliary tool; Allocating the data to be annotated to the at least one terminal device to enable the at least one subject to annotate the data to be annotated based on the annotation template and the annotation auxiliary tool specifically includes: Call offline auxiliary tools; Preprocessing the data to be labeled by the offline auxiliary tool to obtain processed data to be labeled, wherein the preprocessing includes: initially labeling the data to be labeled and generating at least one of labeling prompt information for the data to be labeled; The processed data to be labeled are distributed to a terminal device used by the at least one subject, so that the at least one subject performs the labeling task based on the labeling template and the processed data to be labeled.

[0012] Optionally, a labeling agent is deployed in the artificial intelligence model; Sending the marking template and the loading link of the online auxiliary tool to a terminal device used by the at least one subject so that the terminal device acquires the marking template, and loads the online auxiliary tool based on the loading link, and the at least one subject performs the marking task based on the marking template and the online auxiliary tool, specifically includes: Sending the annotation template and the loading link of the online auxiliary tool to the terminal device used by the at least one subject, and inputting the query information sent by the at least one subject through the online auxiliary tool during the annotation task execution process into the annotation agent to obtain the reply information output by the annotation agent; The reply information is sent to a terminal device used by the at least one subject, so that the at least one subject performs the marking task based on the reply information and the marking template.

[0013] Optionally, before determining at least one subject for performing the labeling task from among the candidate subjects according to the task portrait and the acquired subject attribute information of each candidate subject, the method further includes: Inputting the data to be labeled into the artificial intelligence model to determine the data portrait corresponding to the data to be labeled through the artificial intelligence model; According to the data portrait, determining whether the data to be labeled meets a preset labeling condition; Determining at least one subject for performing the labeling task from among the candidate subjects according to the task portrait and the acquired subject attribute information of each candidate subject, specifically includes: When it is determined that the data to be annotated meets the annotation condition, at least one subject for executing the annotation task is determined from the candidate subjects according to the task portrait and the acquired subject attribute information of each candidate subject.

[0014] Optionally, the method further comprises: When it is determined that the data to be annotated does not meet the annotation condition, a prompt message is sent to the user to prompt the user to resend the data to be annotated.

[0015] Optionally, a distribution agent is deployed in the artificial intelligence model; Determining, by means of the artificial intelligence model, a task profile for describing the labeling task corresponding to the labeling request, and determining, according to the task profile and the acquired subject attribute information of each candidate subject, at least one subject for performing the labeling task from among the candidate subjects, specifically includes: Through the assigned intelligent agent deployed in the artificial intelligence model, a task portrait for describing the labeling task corresponding to the labeling request is determined, and based on the task portrait and the acquired subject attribute information of each candidate subject, at least one subject for performing the labeling task is determined from the candidate subjects.

[0016] Optionally, the method further comprises: Acquire task evaluation information, where the task evaluation information is generated when the at least one subject performs the labeling task, and the task evaluation information includes: at least one of a task log and behavior data, where the behavior data is monitored through a preset tracking point when the at least one subject performs the labeling task; Inputting the task evaluation information into the artificial intelligence model to determine, through the artificial intelligence model, whether there is a problem when the at least one subject performs the labeling task, and generating warning information when it is determined that there is a problem when the at least one subject performs the labeling task; A task warning is performed according to the warning information.

[0017] Optionally, inputting the task evaluation information into the artificial intelligence model to determine, through the artificial intelligence model, whether there is a problem when the at least one subject performs the labeling task, specifically includes: Inputting the task evaluation information and the quality inspection scope information into the artificial intelligence model, so that the artificial intelligence model determines the object to be inspected from the objects involved when the at least one subject performs the labeling task according to the quality inspection scope information, and determines whether there is a problem with the object to be inspected when the at least one subject performs the labeling task according to the task evaluation information; When determining that the at least one subject has a problem in performing the labeling task, generating warning information specifically includes: When it is determined that there is a problem with the object to be inspected when the at least one subject performs the labeling task, early warning information for the object to be inspected is generated.

[0018] Optionally, determining whether there is a problem with the object to be inspected when the at least one subject performs the labeling task according to the task evaluation information specifically includes: Determine a quality inspection tool for the object to be inspected; The quality inspection tool is called to determine, based on the task evaluation information, whether there is a problem with the object to be quality inspected when the at least one subject performs the labeling task.

[0019] Optionally, a quality inspection agent is deployed in the artificial intelligence model; Inputting the task evaluation information into the artificial intelligence model to determine, through the artificial intelligence model, whether there is a problem when the at least one subject performs the labeling task, and generating early warning information when it is determined that there is a problem when the at least one subject performs the labeling task, specifically includes: The task evaluation information is input into the artificial intelligence model to determine whether there are problems when at least one subject performs the labeling task through the quality inspection agent deployed in the artificial intelligence model, and generate early warning information when it is determined that there are problems when at least one subject performs the labeling task.

[0020] According to a second aspect of one or more embodiments of this specification, a device for performing a labeling task is provided, wherein an artificial intelligence model is deployed in the device, including: A first acquisition module is used to obtain a user's annotation request; A determination module, used to determine the user's data to be annotated and annotation requirement information according to the annotation request, wherein the annotation requirement information is used to indicate the user's annotation requirement for the data to be annotated; An input module, used to input the labeling requirement information into the deployed artificial intelligence model, so as to determine, through the artificial intelligence model, a task profile for describing the labeling task corresponding to the labeling request, and determine at least one subject for performing the labeling task from among the candidate subjects according to the task profile and the subject attribute information of each candidate subject obtained; The allocation module is used to allocate the data to be labeled to the terminal device used by the at least one subject to perform the labeling task.

[0021] According to a second aspect of one or more embodiments of the present specification, an electronic device is proposed, comprising: a processor; a memory for storing processor executable instructions; wherein the processor implements the steps of the above-mentioned labeling task execution method by running the executable instructions.

[0022] According to a second aspect of one or more embodiments of the present specification, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the above-mentioned labeling task execution method are implemented.

[0023] According to a second aspect of one or more embodiments of the present specification, a computer program product is proposed, including a computer program / instruction, which implements the steps of the above-mentioned labeling task execution method when executed by a processor.

[0024] It can be seen from the above embodiments that the task requirements and characteristics of the labeling task can be effectively described according to the determined task portrait. Therefore, the artificial intelligence model can automatically determine the subject suitable for labeling the data to be labeled through the task portrait. The whole process not only greatly reduces human intervention and reduces the consumption of manpower costs, but also ensures that the data to be labeled and the subject are highly adapted. This not only greatly improves the allocation efficiency in the task allocation stage, but also further improves the labeling efficiency in the data labeling stage. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A flowchart of a method for executing a labeling task provided in this specification; Figure 2 A detailed process diagram of detecting the data to be labeled based on the data portrait provided in this specification to complete the entire labeling task; Figure 3 A schematic diagram of embedding or calling multiple agents to perform a labeling task provided in this specification; Figure 4 A diagram of the annotation system architecture with three agents deployed for this specification; Figure 5 A quality inspection process diagram provided for this manual; Figure 6 A schematic diagram of the structure of a device provided in this manual; Figure 7 This is a block diagram of a labeling task execution device provided in this specification. DETAILED DESCRIPTION

[0026] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0027] At present, in order to ensure that artificial intelligence models can effectively and accurately perform services such as intelligent customer service, unmanned driving, information recommendation, smart medical care, data prediction, etc., it is necessary to continuously use a large number of samples to train artificial intelligence models.

[0028] During the data labeling process, the user needs to first deliver the data to be labeled (referred to as the data to be labeled in this manual) to the labeling platform. The labeling platform needs to manually analyze this data through experience to determine the labeling personnel that are suitable for these data to be labeled. Then, the data to be labeled is assigned to these determined labeling personnel, and these labeling personnel complete the data labeling.

[0029] However, since the data to be labeled is currently assigned to the labelers through manual analysis, it is bound to consume a lot of labor costs. In addition, due to the influence of human subjective factors, there may be a mismatch between the data to be labeled and the determined labelers, which makes the determined labelers unable to adapt to the data after receiving it, and thus unable to complete the labeling task. Not only that, the labelers can only rely on their own experience after receiving the data to be labeled, which not only leads to low overall labeling efficiency, but also may cause labeling errors and low labeling quality.

[0030] In order to solve the above problems, this specification provides a method for executing a labeling task. According to the determined task portrait of the labeling task, the task requirements and characteristics of the labeling task can be effectively described. Therefore, the subject suitable for labeling the data to be labeled can be determined through the method provided in this specification. The whole process not only greatly reduces human intervention and reduces the consumption of manpower costs, but also ensures that the data to be labeled and the subject are highly adapted, thereby significantly improving the efficiency of data labeling as a whole.

[0031] Moreover, the identified subjects can use various annotation auxiliary tools to complete the annotation task, which greatly improves the efficiency of completing the annotation task. With the assistance of the annotation auxiliary tools, the quality of completing the annotation task is significantly improved.

[0032] The following important concepts are involved in this manual: Artificial intelligence model: an artificial intelligence model trained through algorithms and data, designed to imitate human intelligence to perform specific tasks or solve complex problems. At present, most artificial intelligence models are built based on deep learning architectures, especially advanced structures such as Transformer. These architectures use multi-layer neural networks to process input data and can capture complex patterns and relationships. For example, the Transformer architecture uses a self-attention mechanism to enable the model to pay attention to the importance of different parts when processing sequence data, thereby significantly improving language understanding and generation capabilities. In addition, convolutional neural networks (CNNs) are good at processing image data, while recurrent neural networks (RNNs) and their variants (such as LSTM and GRU) are suitable for processing time series data or contextual information in text.

[0033] In addition, AI models usually have a large number of parameters, which enables them to learn more detailed language and pattern features. Large language models are a typical example. They use billions or more parameters and are trained on massive amounts of text data to generate high-quality natural language content and understand the nuances of complex contexts.

[0034] In practical applications, artificial intelligence models have shown a wide range of functions and can perform tasks such as image classification, speech recognition, predictive analysis, trajectory planning, etc., thus bringing great convenience to users' actual lives. In this specification, artificial intelligence models may refer to large language models.

[0035] Agent: The agent in this specification refers to a complex software program with a dedicated purpose and capable of providing specific functions. It can perceive information, make decisions and perform corresponding operations in a dynamic environment based on preset rules or learned knowledge. Agents can work independently or collaborate with other agents or human users to complete complex tasks.

[0036] In fact, the intelligent agent can be regarded as a smaller-scale artificial intelligence model, which can be obtained through sample training. However, due to the significant differences in the number of network layers and the number of parameters from the artificial intelligence model, the intelligent agent can often only provide a single function. Accordingly, the samples used by the intelligent agent are also samples that specifically serve this single function.

[0037] For example, in practical applications, large language models have multiple capabilities such as text response, image recognition, speech recognition, and multimodal information recognition, so it is often necessary to use rich multimodal samples such as text, pictures, and audio to train large language models. If you want to get an agent specifically for image generation, you only need to use images and texts used to describe images as samples to get the agent.

[0038] Tools: The tools provided in this manual refer to software programs with certain functions. Tools can be program packages that integrate various rules, or they can be small dedicated models built using artificial intelligence technology similar to the above-mentioned intelligent agents. The tools in this manual are mainly used to further assist the functions to be realized by the intelligent agents, and provide data support and functional support for the intelligent agents.

[0039] The relationship between AI models, agents, and tools: As for the relationship between these three, the intelligent agent can embed or call the above-mentioned artificial intelligence model, the purpose of which is to use the powerful natural language processing capabilities of the artificial intelligence model to enhance the functions of the intelligent agent. In other words, after analyzing and processing the information input by the user, the artificial intelligence model can provide the intelligent agent with input information that is easy for the intelligent agent to understand. Correspondingly, the intelligent agent can output output results that meet the actual needs of the user based on the input information given by the artificial intelligence model. The functions provided by the intelligent agent may also need to rely on other auxiliary tools to be implemented. That is, in addition to obtaining the input information processed by the artificial intelligence model, the intelligent agent may need to call other tools to further process the input information in the process of processing and analyzing the input information to obtain the final output result, or obtain some auxiliary data by calling other tools, and then obtain the final output result by combining the input information and auxiliary information.

[0040] In addition, for the solution provided in this specification, the above-mentioned intelligent agent may not be used. That is, if the artificial intelligence model has the functions required to implement this solution, there is no need to embed or call the intelligent agent additionally. Through the artificial intelligence model and the above-mentioned tools, all requirements involved in the execution of the labeling task can be achieved.

[0041] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0042] Figure 1 A flowchart of a method for executing a labeling task provided in this specification.

[0043] S100: Obtaining a user's annotation request.

[0044] In this specification, the user can send the data to be annotated in the annotation request to the annotation platform, and the annotation platform will complete the subsequent annotation tasks.

[0045] The labeling platform is equipped with a labeling system, which can be understood as a device cluster consisting of at least one server and several terminal devices. Users who need data labeling, operators in the labeling platform, or entities that perform labeling tasks in the labeling platform can collaborate with each other through the labeling system and divide the work to complete different task stages of the entire labeling task.

[0046] The user mentioned above may be a single individual or a demander of data annotation services. The demander may be an external company of the annotation platform where the annotation system is located, or an internal staff member or internal department of the annotation platform where the annotation system is located.

[0047] When the demander belongs to an external company, the above-mentioned annotation system is provided with an external interface, through which the demander can access the annotation platform where the annotation system is located and query the various services provided by the annotation platform. The demander can also send an annotation request to the annotation system of the annotation platform through the external interface, and the annotation system will perform the corresponding annotation task.

[0048] When the demander is an internal staff member or an internal department of the annotation platform, the above-mentioned annotation system is also provided with an internal interface, through which the demander can send an annotation request to the annotation system, and the annotation system coordinates and assigns the annotation personnel to perform the annotation task and perform the corresponding annotation task.

[0049] S102: Determine the data to be annotated and the annotation requirement information of the user according to the annotation request, where the annotation requirement information is used to indicate the annotation requirement of the user for the data to be annotated.

[0050] The annotation system can obtain the user's data to be annotated based on the above annotation request. The data to be annotated can be carried in the annotation request, and the annotation system obtains the data to be annotated by parsing the annotation request.

[0051] Of course, the annotation system can also establish a data transmission channel with the device used by the user through the annotation request, and obtain the user's data to be annotated through the data transmission channel. In short, this specification does not clearly limit the specific way in which the annotation system obtains the data to be annotated.

[0052] The above-mentioned data to be labeled can be in the form of data such as text, pictures, videos, audio, etc. The specific form of the data to be labeled is closely related to the business function of the artificial intelligence model that the user wants to train.

[0053] For example, if the artificial intelligence model that the user wants to train is mainly used to identify targets in pictures, the data to be labeled can be data in the form of pictures; for another example, if the artificial intelligence model that the user wants to train is a large language model, which is mainly used to realize question-and-answer functions, the data to be labeled can be dialogue text in text form.

[0054] It should be noted that the data to be labeled is actually multiple data that need to be labeled. These data may all be unlabeled data, or may contain some unlabeled data and some roughly labeled data. Therefore, when the labeling system performs the labeling task, it is necessary to label the unlabeled data and re-label the roughly labeled data.

[0055] Furthermore, in this specification, the user can also send annotation requirement information to the annotation platform. The annotation requirement information mainly describes the user's annotation requirements. For example, the annotation requirement information can be used to describe the application scenario of the data to be annotated, the task magnitude (i.e., the amount of data to be annotated), the annotation mode (whether the annotation needs to be completed by a single person or multiple people, and if there are multiple people, how many people are needed to complete it), whether the data to be annotated needs to be strictly kept confidential, etc., which will not be explained in detail here. In short, the description of the user's needs in the annotation requirement information can reflect the specific situation of the annotation task, the amount of annotation cost that the user can bear, and the annotation quality required by the user.

[0056] It should be noted that in this manual, the user can send the annotation requirement information to the annotation system through the preset interface of the annotation system, or can first send the annotation requirement information to the operator of the annotation platform, and then the operator can enter it into the annotation system through the terminal device used.

[0057] S104: Input the labeling requirement information into the deployed artificial intelligence model to determine, through the artificial intelligence model, a task profile for describing the labeling task corresponding to the labeling request, and determine at least one subject for performing the labeling task from among the candidate subjects based on the task profile and the acquired subject attribute information of each candidate subject.

[0058] After obtaining the above-mentioned data to be annotated and the annotation requirement information, the annotation system can analyze the data to be annotated, combine the analysis results with the annotation requirement information, and determine the task profile corresponding to the subsequent annotation task that needs to be performed.

[0059] The task portrait is mainly used to describe the task requirements and characteristics of the subsequent labeling tasks that need to be performed. This task portrait can reflect the task type of the labeling task, the field to which the data to be labeled belongs, the quality labeling required for the labeling task (what is the required labeling accuracy, etc.), the task type of the labeling task (such as identifying entities in text content, identifying targets in pictures, rewriting text, etc.), task completion time and other key information.

[0060] In this specification, an artificial intelligence model is deployed in the annotation system, so the annotation system can input the annotation requirement information into the artificial intelligence model, and the artificial intelligence model can determine the above-mentioned task portrait by analyzing the annotation requirement information. Among them, the artificial intelligence model can identify the information in the annotation requirement information, determine the task type, field and other information of the annotation task, and thus obtain the task portrait.

[0061] Of course, in the process of determining the task portrait, the labeling system can also input at least part of the data to be labeled together with the labeling requirement information into the artificial intelligence model. The artificial intelligence model can extract key information from the data to be labeled (for example, when the data to be labeled is text data, it can extract words with higher frequency of occurrence to determine the subdivided knowledge field to which the data to be labeled belongs, determine the number of labeling times required for subsequent labeling tasks through the number of data to be labeled, and determine the data type of the data to be labeled through the data form of the data to be labeled, etc.), and then determine the task portrait corresponding to the labeling task based on the key information extracted from the data to be labeled and the key information determined based on the labeling requirement information.

[0062] The artificial intelligence model can further determine the subject used to perform the labeling task based on the determined task portrait. Since the artificial intelligence model has already determined the task portrait that describes the task requirements and characteristics of the labeling task, it is necessary to further obtain the subject attribute information of each candidate subject, and then determine at least one subject that is suitable for the labeling task by matching the task portrait with the subject attribute information of each candidate subject.

[0063] The above subject attribute information is mainly used to describe the characteristics of the candidate subject. For example, it can be used to describe the knowledge field that the candidate subject is good at, the annotation efficiency of the candidate subject, the annotation accuracy of the candidate subject, etc. Therefore, the artificial intelligence model actually finds the subject that meets the characteristics of the annotation task based on the subject characteristics described by the subject attribute information of each candidate subject.

[0064] In this specification, the artificial intelligence model can generate a call instruction to call the preset subject management tool, and the called subject management tool will provide the artificial intelligence model with the subject attribute information of each candidate subject it manages. Among them, the artificial intelligence model can actually automatically produce a task workflow, which is mainly used to reflect the various task actions that the artificial intelligence model needs to perform and the order in which these task actions are executed. To this end, when the annotation system inputs the artificial intelligence model into the annotation requirement information, the artificial intelligence model can trigger the generation of a task workflow based on the annotation requirement information, and perform actions such as task portrait generation and task assignment according to the task workflow. When the task portrait generation action is triggered, the above-mentioned call instruction can be generated, and then the call instruction is sent to the preset call interface to call the above-mentioned subject management tool to synchronously obtain the subject attribute information of each candidate subject while generating the task portrait.

[0065] It should be noted that the specific generation time of the above-mentioned call instruction can be multiple. In addition to the above-mentioned generation when the artificial intelligence model triggers the execution of the task portrait generation action, it can also be generated directly after the artificial intelligence model receives the above-mentioned task requirement information, or it can be generated after the task portrait is generated. In short, this specification does not limit the specific generation time of the call instruction.

[0066] The subject management tools mentioned above can be understood as software programs used to manage or process the subject attribute information of the subject. After receiving the above-mentioned call instructions, each subject management tool can read the subject attribute information of each candidate subject from the preset database and return it to the artificial intelligence model. It can also read the subject attribute information of each subject from the database and further process the information (such as data format conversion, screening out useful fields, etc.), and then return the processed subject attribute information to the artificial intelligence model.

[0067] In this specification, the subject attribute information of each candidate subject obtained by the artificial intelligence model can be the subject attribute information of the full subject directly read from the database through the subject management tool, that is, each full subject is regarded as a candidate subject, or the subject information of the subject suitable for the knowledge field corresponding to the annotation task can be read from the database. Of course, in order to further improve the efficiency of task allocation, each subject can also be screened first, and the screened subjects can be used as candidate subjects to complete the task allocation.

[0068] For the latter method, the key to its execution lies in the effective reuse of historical annotation tasks. In actual applications, the annotation platform has actually undertaken a large number of annotation tasks in history. However, the current annotation platform does not effectively manage these historical annotation tasks, making it difficult for the entities that perform annotation tasks in the annotation platform to effectively reuse these historical annotation tasks to perform the annotation tasks newly undertaken by the annotation platform, which further leads to the current low overall annotation efficiency.

[0069] However, these historical annotation tasks actually have high reuse value. If these historical annotation tasks can be reused, the overall annotation efficiency can be significantly improved, thereby further reducing labor costs.

[0070] Therefore, in this specification, a historical annotation task matching the above annotation task can be determined from various historical annotation tasks as a reference annotation task, and the reference annotation task can be determined based on the above task portrait. Then, the annotation system can determine the subject that has historically performed the reference annotation task as a candidate subject, and obtain the subject attribute information of these candidate subjects, and then determine at least one subject for performing the annotation task according to the subject attribute information of the candidate subject.

[0071] The annotation system can predetermine the task profile of each historical annotation task (e.g., annotate each historical annotation task with various labels that reflect the task profile), and save them in correspondence with each historical annotation task. On this basis, the determined task profile can be matched with the task profile corresponding to each historical annotation task to determine the above-mentioned reference annotation task.

[0072] In the specific process of determining the subject used to perform the labeling task, the task difficulty level corresponding to the labeling task can be determined based on the above-mentioned task portrait, and then, based on the task difficulty level and the subject attribute information of each candidate subject obtained, at least one subject used to perform the labeling task is determined from each candidate subject. The task difficulty level here is used to reflect the task difficulty of the labeling task, which is specifically related to the characteristics and requirements of the task. For example, if it is determined based on the task portrait that the number of data to be labeled is higher than the preset number threshold, and the amount of information contained in the data to be labeled is higher than the preset amount of information, then it can be determined that the task difficulty is high, otherwise, it can be determined that the task difficulty is low. That is, the difficulty of the task can be positively correlated with the number of data to be labeled and the amount of information contained in the data to be labeled.

[0073] In addition, the task profile can be combined with the task priority to determine the difficulty level of the labeling task. The higher the task priority, the higher the task difficulty (the difficulty of this task is mainly considered from the time perspective. The higher the task priority, the tighter the deadline of the task and the higher the required labeling volume per unit time.

[0074] It should be pointed out that the above examples of determining the task difficulty level are only illustrative. In fact, the task difficulty level is also correlated with other dimensions in the task portrait, so we will not give examples one by one here.

[0075] On this basis, the subject attribute information in the candidate subject may also record a field for indicating the difficulty level of the task that the candidate subject is suitable for. The artificial intelligence model can use this to determine at least one subject from each candidate subject to perform the labeling task.

[0076] The subject for performing the labeling task mentioned in this specification may be a labeler for a specific labeling task, or may be a department that performs the labeling task, which includes several labelers.

[0077] In other words, the labeling system can directly determine the labeling personnel who are suitable for performing the labeling task, and can also determine the department that is suitable for performing the labeling task. As for how the department allocates the data to be labeled to the labeling personnel in the department, it can be determined by the department based on actual conditions.

[0078] S106: Distribute the data to be annotated to a terminal device used by the at least one subject to perform the annotation task.

[0079] After the labeling system determines the subject for performing the labeling task through the artificial intelligence model, it will distribute the data to be labeled to the terminal device used by at least one of the determined subjects, so that these subjects can perform the labeling task. The terminal devices used by these subjects can be electronic devices such as desktop computers, laptops, etc., or mobile devices such as tablet computers that can meet the requirements of the labeling task. The embodiments of this specification do not limit the specific form of the terminal devices.

[0080] In order to further improve the efficiency of the subject in performing the annotation task, in this specification, the annotation system can determine the annotation template and annotation auxiliary tools corresponding to the annotation task through the artificial intelligence model based on the task portrait, and recommend them to these subjects. Among them, the annotation template is mainly used to provide a reference to the subject performing the annotation task, indicating how to annotate the annotated data. For example, if the data to be annotated is a picture, and the annotation task is to annotate the specified target object in the picture, then the annotation template can be a picture containing the specified target object, and the position of the specified target object in the picture is clearly marked by the annotation box in the picture; for another example, if the data to be annotated is text data, and the annotation task is to extract the core content of the text to be annotated, then the annotation template can be an original text and the extracted core content, wherein the extracted core content is displayed in brackets at the end of the original text.

[0081] As for the annotation auxiliary tools mentioned above, they are mainly used to provide various convenient annotation functions for the subject who is determined to perform the annotation task, such as text bolding, keyword recognition, knowledge question answering, knowledge retrieval, etc. For this reason, the annotation system can store a variety of annotation auxiliary tools with different functions. The annotation system can select the annotation auxiliary tools suitable for performing the annotation task based on the determined task profile and provide them to the subject who performs the annotation task.

[0082] In this specification, annotation auxiliary tools can be divided into two categories, one is online auxiliary tools and the other is offline auxiliary tools. For online auxiliary tools, it can be understood as an auxiliary tool used online. When the subject performs the annotation task, the subject can interact with the annotation auxiliary tool in real time to provide the subject with services such as knowledge retrieval and annotation question answering.

[0083] In the specific implementation process, the annotation system can directly send the annotation template to the terminal device used by the determined subject, and for the annotation auxiliary tool, the annotation system can send the prompt information used to prompt the subject to actively use the annotation auxiliary tool to the terminal device used by the subject, and the subject can call or load the annotation auxiliary tool from the tool library based on the prompt information. The annotation system can also send the loading link of the annotation auxiliary tool to the terminal device used by the determined subject, and the terminal device can automatically load the corresponding annotation auxiliary tool based on the loading link, without the subject actively completing the loading of the annotation auxiliary tool.

[0084] In order to further improve the efficiency of the subject in performing the labeling task, this manual also provides an offline auxiliary tool, which is mainly used to perform preliminary labeling processing on the data to be labeled. The data actually labeled by the subject may be the data to be labeled after being processed by the offline auxiliary tool.

[0085] To this end, in this specification, the annotation system can call an offline auxiliary tool, and preprocess the data to be annotated through the called offline auxiliary tool to obtain the processed data to be annotated. The preprocessing mentioned here can include various methods, such as initial annotation of the data to be annotated (such as text highlighting, text rewriting, etc.), generating annotation prompt information for the data to be annotated (such as knowledge prompts, etc.), etc.

[0086] Therefore, the above-mentioned offline auxiliary tools can be used for the initial labeling of the data to be labeled, while the online auxiliary tools used by the subject later are used to complete the precise labeling of the data to be labeled based on the initial labeling.

[0087] It can be seen from the above content that the method provided in this specification can be used to determine the subject that is suitable for labeling the data to be labeled. The entire process not only greatly reduces human intervention and reduces the consumption of manpower costs, but also ensures that the data to be labeled and the subject are highly adapted, thereby significantly improving the efficiency of data labeling as a whole.

[0088] Moreover, the identified subjects can use various annotation auxiliary tools to complete the annotation task, which greatly improves the efficiency of completing the annotation task. With the assistance of the annotation auxiliary tools, the quality of completing the annotation task is significantly improved.

[0089] In this specification, in the process of executing the labeling task, corresponding task logs and behavior data will be generated. The so-called behavior data refers to the data monitored by the preset embedded points when the subject performs the labeling task, which is mainly used to reflect the actual operation of the subject in the labeling process. Therefore, these task logs and behavior data can reflect the actual execution of the labeling task from multiple angles. To this end, the labeling system can obtain the task evaluation information of the labeling task, which includes the task log and behavior data. Then, based on the task evaluation information, it can be judged whether there is a problem in the subject's execution of the labeling task.

[0090] The above-mentioned task evaluation information can record information such as: task name, task ID, start time of annotation, actual daily capacity (reflecting the number of data actually annotated by the subject every day), annotation task completion rate, the average time the mouse pointer stays in the same position during the subject annotation process, the average time of single data annotation during the subject annotation process, and other information.

[0091] During the specific implementation process, the labeling system can input the task evaluation information into the above-mentioned artificial intelligence model to determine whether there is a problem when at least one subject performs the labeling task through the artificial intelligence model, and generate warning information when it is determined that there is a problem in the process of at least one subject performing the labeling task, and then perform task warning through the warning information.

[0092] Among them, during the quality inspection stage, the annotation system can also obtain quality inspection scope information, which is used to indicate the objects that need to be inspected, which can be referred to as objects to be inspected in the subsequent content. Objects to be inspected can be any objects involved when the subject performs the annotation task, such as recommended tools (including the online auxiliary tools and offline auxiliary tools mentioned above), the subject determined to perform the annotation task, the task itself, and the supplier who performs the annotation task (in actual applications, the annotation platform can entrust the annotation task to an external supplier based on business needs), etc.

[0093] After determining the above-mentioned quality inspection scope information, the labeling system can input the above-mentioned task evaluation information and quality inspection scope information into the artificial intelligence model together, so that the artificial intelligence model can determine the objects to be inspected from the objects involved when the subject performs the labeling task according to the quality inspection scope information, and determine whether there are any problems with the objects to be inspected when the subject performs the labeling task according to the task evaluation information.

[0094] When it is determined that there are problems with the object to be inspected, the artificial intelligence model can be used to generate early warning information specifically for the object to be inspected, so that the operators of the labeling platform can be reminded to solve the problems in a timely manner through the early warning information.

[0095] Furthermore, in actual applications, a variety of quality inspection tools can be maintained in the annotation system, and different quality inspection tools are used to detect different types of problems. Therefore, after determining the above-mentioned object to be inspected, the artificial intelligence model can determine the quality inspection tool suitable for the object to be inspected, and then call the quality inspection tool to determine whether there is a problem with the object to be inspected when the subject performs the annotation task based on the task evaluation information.

[0096] It can be seen from the above process that the artificial intelligence model in this manual plays a core role in the task allocation stage, the labeling task execution stage, and the task quality inspection stage. Through the powerful analysis ability of the artificial intelligence model, the various task stages are connected in series, and different functional services are provided according to the characteristics of different stages, thereby greatly improving the task execution efficiency of the entire process from user entrustment of labeling tasks to the final labeling task delivery.

[0097] The above content has generally described the various task links involved in the entire annotation task execution process. There are also some other contents involved in these task links, which will be explained in detail below.

[0098] In actual applications, there may be certain problems with the data to be labeled, such as missing data, data duplication, etc. In order to ensure that the labeling system can accurately determine the above-mentioned task portrait and ensure the smooth execution of subsequent labeling tasks, in an embodiment of the present specification, the labeling system can first determine the data portrait corresponding to the data to be labeled based on the data to be labeled.

[0099] The data portrait is mainly used to describe the data characteristics of the data to be labeled, including the number of data to be labeled, the type of data to be labeled, the knowledge field corresponding to the data to be labeled, etc. At the same time, it also contains information used to reflect whether there are various problems with the data to be labeled, such as information used to reflect whether there is data missing in the data to be labeled, and information used to reflect whether there is data duplication in the data to be labeled.

[0100] Therefore, the annotation system can detect the annotated data based on the data portrait, and perform subsequent processes based on the detection results, such as Figure 2 shown.

[0101] Figure 2 A detailed process diagram of detecting the data to be labeled based on data profiling to complete the entire labeling task is provided in this manual.

[0102] After the annotation system obtains the user's data to be annotated, it can first determine the corresponding data portrait based on the data to be annotated, and then perform data analysis on the data to be annotated based on the data portrait to determine whether the data to be annotated meets the annotation conditions. The annotation system can determine the data portrait through the above-mentioned artificial intelligence model, that is, input the data to be annotated into the artificial intelligence model, and the artificial intelligence model outputs the corresponding data portrait.

[0103] If it is determined that the data to be labeled does not have problems such as missing data, data duplication, etc., then it can be determined that the data to be labeled meets the labeling conditions, and then the task profile can be determined, and based on the task profile, the labeling template used for the labeling task can be recommended.

[0104] Afterwards, the labeling system can create the labeling task and further determine the task difficulty level of the labeling task. After determining the task difficulty level, the historical labeling tasks can be matched based on the task difficulty level (or combined with the task profile), and at least one subject for performing the labeling task can be recommended according to the matching results to allocate the data to be labeled.

[0105] If it is determined that the data to be annotated has problems such as missing data, data duplication, etc., a prompt message may be sent to the user to prompt the user to resend the data to be annotated.

[0106] For the above prompt message, the specific prompt content also corresponds to the problem of the data to be labeled. For example, if it is determined that there is data missing in the data to be labeled, the user can be prompted to resubmit the complete data.

[0107] Of course, for data to be annotated that have problems, the annotation system can also repair these data to be annotated on its own without prompting the user. For example, in the case of data duplication, the annotation system can identify the duplicate data to be annotated, and then perform deduplication operations on these data to be annotated; for another example, in the case of missing data, the annotation system can use the semantic recognition ability of the artificial intelligence model to identify the data to be annotated that has missing data content, and then, through the semantic analysis of the existing data content in these data to be annotated that has missing data content, complete the missing content, and thus obtain the completed data to be annotated.

[0108] It can be seen from the above content that there is some overlap between the data portrait and the task portrait. In fact, the task portrait can contain all the information of the data portrait. In addition, the task portrait can also contain other information used to focus on reflecting the task characteristics of the labeling task. Therefore, in the embodiments of this specification, the relationship between the task portrait and the data portrait can actually be that the task portrait contains the data portrait.

[0109] in addition, Figure 2 The allocation method verification is also mentioned in this paper. This process is mainly used to detect whether the allocation method of the labeling task is reasonable through a specified method. For example, it can be manually detected by some designated personnel to vote to determine whether the allocation method is feasible based on the voting results.

[0110] In this specification, multiple agents can be embedded or called in the artificial intelligence model. Different agents are responsible for different task stages of the entire labeling task and provide matching functions for different task stages, such as Figure 3 shown.

[0111] Figure 3 A schematic diagram for embedding or calling multiple agents to perform labeling tasks provided in this specification.

[0112] exist Figure 3 As can be seen from the figure, there are three types of agents in total, namely allocation agent, labeling agent and quality inspection agent.

[0113] The allocation agent is used to allocate labeling tasks. In this stage, the operator of the labeling platform can undertake the labeling tasks entrusted by users. Users need to send the data to be labeled and the task requirement information to the operator, and the operator can input the data to be labeled and the task requirement information into the artificial intelligence model. The artificial intelligence model can extract key information from the data to be labeled and the task requirement information through its powerful natural language processing capabilities, and process the key information to input the processed information into the allocation agent embedded in the artificial intelligence model or the called allocation agent.

[0114] The assignment agent can determine tasks such as task profiles, recommended annotation templates, and the labelers (i.e. Figure 3 The labeling personnel are the subjects mentioned above) and labeling strategies (labeling strategies can be used to reflect the allocation strategies of the subjects who perform labeling tasks, as well as the strategies when the subjects perform labeling tasks) and other information, and return this information to the terminal device used by the operator.

[0115] The operator can complete the labeling task assignment by using the information given by the assigned agent. When the labeler is performing the labeling task, the labeling agent can use the information given by the assigned agent (such as task profile, labeling strategy, etc.). Figure 3 In the figure, the assignment agent is represented by a dotted line pointing to the annotation agent, and the online auxiliary tools recommended to the annotation personnel are determined from the preset tool store. In addition, the annotation agent can reply to the corresponding reply information based on the query information sent by the annotation personnel when interacting with the annotation agent in real time, so as to perform services such as knowledge question and answer, open domain retrieval (retrieve the knowledge required by the annotation personnel from the external knowledge base), etc.

[0116] The main function of the quality inspection agent is to analyze whether there are any problems in the labeling task performed by the labeler based on the acquired task logs, behavior data, etc. Among them, the problems may be related to the labeler or not, which needs to be determined through the task evaluation information.

[0117] For example, if it is determined through task evaluation information that the labeling personnel's daily production capacity is significantly lower than the daily planned production capacity, it can be determined that the labeling personnel have a problem of low labeling efficiency. For another example, if it is determined through task evaluation information that a network failure occurred during the labeling task, it can be determined that a network problem occurred during the labeling task execution.

[0118] For another example, if according to the remark information contained in the task evaluation information (the remark information may be generated by the annotation personnel performing the remark operation in the annotation system based on the actual situation during the execution of the annotation task, and the remark information may be recorded in the task log), it is determined that the environment in which the terminal device used by the annotation personnel is located does not meet the required environmental requirements and is in the process of being transformed, then it can be determined that an environmental mismatch problem occurred when the subject performed the annotation task.

[0119] For another example, in actual applications, after the labeler completes the labeling task, the labeled data needs to be further quality inspected to determine the labeling accuracy (this process can be completed by a dedicated quality inspector or automatically by dedicated equipment). Therefore, if it is determined based on the task evaluation information that the labeling accuracy corresponding to the data labeled by the labeler is lower than the preset accuracy, it can be determined that the labeler has an inaccurate labeling problem when performing the labeling task.

[0120] Other situations are not described here one by one.

[0121] In addition to the task logs and behavior data provided by the annotation system, the annotation agent can also provide the required data for the quality inspection agent to perform quality inspection. For example, if the annotation agent itself has certain statistical functions, the annotation agent can count the number of times the annotation personnel call online auxiliary tools during the annotation task, and send the counted number to the quality inspection agent. If the number exceeds the preset number, the quality inspection agent can determine that the annotation personnel may not be well-matched with the data to be annotated, so it is necessary to call the online auxiliary tools multiple times, and the annotation efficiency is low. On the contrary, it can be determined that the annotation personnel are well-matched with the data to be annotated, and there is no need to call the online auxiliary tools multiple times during the annotation task, and the annotation efficiency is high.

[0122] The above examples mainly involve Figure 3 In addition, the quality inspection agent can also use other functions such as standard verification (verifying the annotation standards used by the annotation personnel) to feed back the quality inspection results to the operation personnel, who can then complete the problem repair based on the warning information contained in the quality inspection results.

[0123] The above is mainly used to reflect the collaborative relationship between the three agents from the execution process of the labeling task. The following will describe the functions and roles provided by the three agents from the perspective of system architecture, such as Figure 4 shown.

[0124] Figure 4 A schematic diagram of the annotation system architecture provided for this specification, which is deployed with three intelligent agents.

[0125] The entire annotation system can be roughly divided into the front-end, middle-end and back-end.

[0126] The front desk is mainly used for operators, so that they can input the data to be labeled and the user's task requirement information into the allocation agent through the front-end interface displayed by the terminal device. The allocation agent then provides task allocation strategies to the operators.

[0127] There are many tools integrated in the middle platform, so the middle platform can be understood as a tool library. The tools in this tool library can be oriented to different task stages, such as, Figure 3 The allocation tool in can be the tool used by the allocation agent when allocating tasks. Figure 3 The intelligent portrait determination in the can be a tool called by the allocation agent in the process of determining task portraits or data portraits. In other words, in addition to using its own capabilities to determine task portraits and formulate task allocation strategies, the allocation agent can also use Figure 3 This can be done using the tools in the tool library.

[0128] The middle platform can also make intelligent tool recommendations, that is, based on the task profile, it can recommend the required online and offline auxiliary tools to the labelers. Once the recommended tools are determined, the tool settings can be made so that the labelers can automatically or actively call these tools to complete the labeling tasks. It can be seen that the functions provided by the middle platform actually cover the above-mentioned labeling agents. Therefore, the labeling agent can be used as a tool and deployed in the middle platform (the labeling agent is in Figure 4 not specifically shown).

[0129] Zhongtaizhong also has a quality inspection agent. By calling the quality inspection agent in Zhongtaizhong, the quality inspection of the labeling task can be realized. Among them, the quality inspection agent can complete the quality inspection by itself according to its own functions, or it can complete the quality inspection by calling the relevant tools in the tool library in Zhongtaizhong.

[0130] A knowledge base is maintained in the background. From the above content, it can be seen that when performing labeling tasks, labelers can complete labeling through real-time interaction with online auxiliary tools (or labeling agents). During this process, if the labelers need to obtain some knowledge required for labeling or knowledge in the knowledge field to which the data to be labeled belongs, the background can output the required knowledge to the labelers through the knowledge base it maintains, so as to assist the labelers in performing the labeling tasks.

[0131] In addition, during the quality inspection process, the labeling system can point out specific problems and treatment strategies to operators through the identified warning information, and can take corresponding measures for the objects with problems, such as Figure 5 shown.

[0132] Figure 5 This is a quality inspection process diagram provided for this manual.

[0133] A tool square is maintained in the annotation system, similar to Figure 4 The tool library in the tool library is just that this tool library is specially used to maintain various tools required for quality inspection, and different tools are used for different types of quality inspection problems.

[0134] exist Figure 5 In the examples shown, the problems can be divided into subject problems, supplier problems, task problems, intelligent tool recommendation problems, etc. Subject problems are mainly used to reflect the abnormal conditions that occur when the subject is performing the labeling task, such as fluctuating status, high fluctuations in daily production capacity, and many missed labelings; supplier problems are mainly used to reflect the various abnormal conditions involved in the task docking when the labeling task is assigned to the supplier for execution; task problems are mainly used to reflect whether the artificial intelligence model is accurate in analyzing task characteristics or requirements, such as whether the task portrait is accurate; intelligent tool recommendation problems are mainly used to reflect whether the artificial intelligence model can accurately and reasonably recommend auxiliary tools that are compatible with the labeling task.

[0135] The data required for quality inspection is maintained by a database, which stores data such as task logs, behavior data, test results, and early warning feedback. The test results mentioned here refer to the fact that before assigning tasks, the labelers determined to perform labeling tasks can be tested to determine whether the labelers meet the labeling requirements. These test results will also be stored in the database. Early warning feedback is used to record the feedback operations performed by operators after receiving early warning information, such as the specific way to deal with the problem, and the feedback on whether the early warning information output by the artificial intelligence model (or quality inspection agent) is accurate.

[0136] During the quality inspection process, it is necessary to provide the AI ​​model (or quality inspection agent) with quality inspection scope information to prompt the AI ​​model with the scope of objects that need to be inspected and the specific quality inspection content. The AI ​​model (or quality inspection agent) needs to further determine the quality inspection tool that is suitable for the object to be inspected based on the quality inspection content and the objects to be inspected indicated in the quality inspection scope information. The determined quality inspection tool contains specific quality inspection rules (such as the threshold required to determine whether there is a problem). Therefore, after the quality inspection tool is determined, the corresponding workflow can be generated, that is, the data that needs to be inspected is input into the quality inspection tool, and the rules in the quality inspection tool output the corresponding quality inspection results.

[0137] The artificial intelligence model (or quality inspection agent) will generate corresponding warning information based on the quality inspection results obtained by calling the quality inspection tool. The warning information contains basic information and specific problem descriptions of the problem, as well as a jump link to the location where the problem occurred. By touching the jump link, you can jump to the data to be labeled where the problem occurred, the task log of the labeler performing the labeling task, etc.

[0138] At the same time, the warning information also includes a disposal strategy for the problem, through which the operator can complete the problem repair or disposal. In addition, the warning information can also include an option or input box for the operator to feedback whether the warning information is accurate. The operator can jump to the link to determine whether the warning information is accurate, thereby completing the feedback.

[0139] In addition, the labeling system is also equipped with a variety of disposal tools. Operators can choose appropriate disposal tools according to specific circumstances to complete the disposal and repair of problems. Different disposal tools have different functions. For example, if it is determined that there is an error in the task profile determined at the beginning of the artificial intelligence model, the corresponding disposal tool can be used to force the labeling to stop; for another example, if the problem is unreasonable task allocation, the corresponding disposal tool can be selected to reallocate tasks; for another example, if it is determined that the labeling efficiency or labeling accuracy of the labeler is reduced, the corresponding disposal tool can be selected to re-evaluate the labeler (i.e. Figure 5 Only after passing the assessment can the labeling personnel carry out the labeling task.

[0140] The above examples are mainly for problems that occur within the annotation platform. As mentioned above, the annotation platform can also assign annotation tasks to external suppliers. Therefore, the annotation system also has corresponding disposal tools for problems that occur with suppliers. For example, when a problem occurs, a warning message can be sent to the client provided by the annotation platform used by the supplier through a pop-up message window; for another example, when a problem occurs, a warning message can be sent to the client provided by the annotation platform used by the supplier through a voice broadcast.

[0141] Figure 6 This is a schematic diagram of the structure of a device provided in this manual. Figure 6 At the hardware level, the device includes a processor 602, an internal bus 604, a network interface 606, a memory 608, and a non-volatile memory 610, and may also include hardware required for other functions. One or more embodiments of this specification may be implemented based on software, such as the processor 602 reading the corresponding computer program from the non-volatile memory 610 into the memory 608 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0142] Please refer to Figure 7 The labeling task execution device provided in this specification can be applied to Figure 6 The device shown in the figure is used to implement the technical solution of this specification. The marking task execution device may include: A first acquisition module 700 is used to acquire a user's annotation request; A determination module 702 is used to determine the user's data to be annotated and annotation requirement information according to the annotation request, wherein the annotation requirement information is used to indicate the user's annotation requirement for the data to be annotated; An input module 704 is used to input the labeling requirement information into the deployed artificial intelligence model, so as to determine, through the artificial intelligence model, a task profile for describing the labeling task corresponding to the labeling request, and determine at least one subject for performing the labeling task from among the candidate subjects according to the task profile and the subject attribute information of each candidate subject obtained; The allocation module 706 is used to allocate the data to be annotated to the terminal device used by the at least one subject to perform the annotation task.

[0143] Optionally, the input module 704 is specifically used to determine the task difficulty level corresponding to the labeling task based on the task portrait; and determine at least one subject from the candidate subjects to perform the labeling task based on the task difficulty level and the acquired subject attribute information of each candidate subject.

[0144] Optionally, the input module 704 is specifically used to, based on the task portrait, determine a historical annotation task that matches the annotation task from various historical annotation tasks as a reference annotation task; use the subject that historically performed the reference annotation task as a candidate subject, and obtain subject attribute information of the candidate subject.

[0145] Optionally, the device further comprises: The second acquisition module 708 is used to call a preset subject management tool through the calling instruction generated by the artificial intelligence model to obtain the subject attribute information of each candidate subject through the called subject management tool. The calling instruction is generated by the artificial intelligence model after receiving the labeling requirement information.

[0146] Optionally, before the allocation module 706 allocates the data to be annotated to the terminal device used by the at least one subject, the input module 704 is further used to determine, based on the task portrait, a labeling template and a labeling auxiliary tool corresponding to the labeling task through the artificial intelligence model; The allocation module 706 is specifically used to allocate the data to be annotated to the at least one terminal device in use, so that the at least one subject performs the annotation task based on the annotation template and the annotation auxiliary tool.

[0147] Optionally, the annotation auxiliary tool includes: an online auxiliary tool; The allocation module 706 is specifically used to send the loading link of the annotation template and the online auxiliary tool to the terminal device used by the at least one subject, so that the terminal device obtains the annotation template, and loads the online auxiliary tool based on the loading link, and the at least one subject performs the annotation task based on the annotation template and the online auxiliary tool.

[0148] Optionally, the annotation auxiliary tool includes: an offline auxiliary tool; The allocation module 706 is specifically used to call an offline auxiliary tool; pre-process the data to be labeled through the offline auxiliary tool to obtain processed data to be labeled, and the pre-processing includes: initially labeling the data to be labeled and generating at least one of labeling prompt information for the data to be labeled; and allocate the processed data to be labeled to a terminal device used by the at least one subject, so that the at least one subject performs the labeling task based on the labeling template and the processed data to be labeled.

[0149] Optionally, a labeling agent is deployed in the artificial intelligence model; The allocation module 706 is specifically used to send the loading link of the annotation template and the online auxiliary tool to the terminal device used by the at least one subject, and input the query information sent by the at least one subject through the online auxiliary tool during the execution of the annotation task into the annotation intelligent agent to obtain the reply information output by the annotation intelligent agent; send the reply information to the terminal device used by the at least one subject, so that the at least one subject performs the annotation task based on the reply information and the annotation template.

[0150] Optionally, before the input module 704 determines at least one subject for performing the labeling task from the candidate subjects according to the task portrait and the acquired subject attribute information of each candidate subject, the input module 704 is further used to input the data to be labeled into the artificial intelligence model to determine the data portrait corresponding to the data to be labeled through the artificial intelligence model; and judge whether the data to be labeled meets the preset labeling condition according to the data portrait; The input module 704 is specifically used to, when it is determined that the data to be annotated meets the annotation condition, determine at least one subject for performing the annotation task from among the candidate subjects according to the task portrait and the acquired subject attribute information of each candidate subject.

[0151] Optionally, the device further comprises: The prompt module 710 is configured to send a prompt message to the user to prompt the user to resend the data to be labeled when the input module 704 determines that the data to be labeled does not meet the labeling condition.

[0152] Optionally, a distribution agent is deployed in the artificial intelligence model; The input module 704 is specifically used to determine, through the assigned agent deployed in the artificial intelligence model, a task portrait used to describe the labeling task corresponding to the labeling request, and to determine at least one subject from the candidate subjects to perform the labeling task based on the task portrait and the acquired subject attribute information of each candidate subject.

[0153] Optionally, the device further comprises: The early warning module 712 is used to obtain task evaluation information, which is generated when the at least one subject performs the labeling task, and the task evaluation information includes: at least one of a task log and behavior data, and the behavior data is monitored through preset tracking points when the at least one subject performs the labeling task; the task evaluation information is input into the artificial intelligence model to determine whether there are problems when the at least one subject performs the labeling task through the artificial intelligence model, and generate early warning information when it is determined that there are problems in the process of the at least one subject performing the labeling task; and task warning is performed according to the early warning information.

[0154] Optionally, the early warning module 712 is specifically used to input the task evaluation information and the quality inspection scope information into the artificial intelligence model, so that the artificial intelligence model determines the object to be inspected from the objects involved when the at least one subject performs the labeling task according to the quality inspection scope information, and determines whether there is a problem with the object to be inspected when the at least one subject performs the labeling task according to the task evaluation information; The warning module 712 is specifically configured to generate warning information for the object to be inspected when it is determined that there is a problem with the object to be inspected when the at least one subject performs the labeling task.

[0155] Optionally, the early warning module 712 is specifically used to determine a quality inspection tool for the object to be inspected; call the quality inspection tool to determine whether there is a problem with the object to be inspected when the at least one subject performs the labeling task based on the task evaluation information.

[0156] Optionally, a quality inspection agent is deployed in the artificial intelligence model; The early warning module 712 is specifically used to input the task evaluation information into the artificial intelligence model so as to determine whether there are problems when at least one subject performs the labeling task through the quality inspection agent deployed in the artificial intelligence model, and generate early warning information when it is determined that there are problems when at least one subject performs the labeling task.

[0157] Based on the same concept as the above method, this specification also provides an electronic device, including: a processor; a memory for storing processor executable instructions; wherein the processor implements the steps of the method described in any of the above embodiments by running the executable instructions.

[0158] Based on the same concept as the above method, this specification also provides a computer-readable storage medium on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in any of the above embodiments are implemented.

[0159] Based on the same concept as the above method, this specification also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any of the above embodiments when executed by a processor.

[0160] This specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0161] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0162] The above are only embodiments of this specification and are not intended to limit this specification. For those skilled in the art, this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification should be included in the scope of the claims of this specification.

Claims

1. A method for executing a labeling task, the method being applied to a labeling system, wherein an artificial intelligence model is deployed in the labeling system, comprising: Get the user's annotation request; Determining the user's data to be annotated and annotation requirement information according to the annotation request, wherein the annotation requirement information is used to indicate the user's annotation requirement for the data to be annotated; Inputting the labeling requirement information into the deployed artificial intelligence model, so as to determine, through the artificial intelligence model, a task profile for describing the labeling task corresponding to the labeling request, and determining at least one subject for performing the labeling task from among the candidate subjects according to the task profile and the subject attribute information of each candidate subject obtained; The data to be labeled is distributed to a terminal device used by the at least one subject to perform the labeling task.

2. The method according to claim 1, determining at least one subject for performing the labeling task from among the candidate subjects according to the task portrait and the acquired subject attribute information of each candidate subject, specifically comprising: Determining a task difficulty level corresponding to the labeling task according to the task profile; At least one subject for performing the labeling task is determined from among the candidate subjects according to the task difficulty level and the acquired subject attribute information of each candidate subject.

3. The method according to claim 1, obtaining the subject attribute information of each candidate subject, specifically comprising: Based on the task portrait, determining a historical annotation task that matches the annotation task from various historical annotation tasks as a reference annotation task; The subject that has historically performed the reference annotation task is used as a candidate subject, and subject attribute information of the candidate subject is obtained.

4. The method according to claim 1, obtaining the subject attribute information of each candidate subject, specifically comprising: The calling instruction generated by the artificial intelligence model calls the preset subject management tool to obtain the subject attribute information of each candidate subject through the called subject management tool. The calling instruction is generated by the artificial intelligence model after receiving the labeling requirement information.

5. The method according to claim 1, before distributing the data to be annotated to the terminal device used by the at least one subject, the method further comprises: Based on the task portrait, determining a labeling template and a labeling auxiliary tool corresponding to the labeling task through the artificial intelligence model; Allocating the data to be annotated to a terminal device used by the at least one subject to perform the annotation task specifically includes: The data to be annotated is distributed to the at least one terminal device used, so that the at least one subject performs the annotation task based on the annotation template and the annotation auxiliary tool.

6. The method according to claim 5, wherein the annotation auxiliary tool comprises: Online assistance tools; Allocating the data to be annotated to the at least one terminal device used so that the at least one subject performs the annotation task based on the annotation template and the annotation auxiliary tool specifically includes: The loading link of the annotation template and the online auxiliary tool is sent to the terminal device used by the at least one subject, so that the terminal device obtains the annotation template, and loads the online auxiliary tool based on the loading link, and the at least one subject performs the annotation task based on the annotation template and the online auxiliary tool.

7. The method according to claim 5, wherein the annotation auxiliary tool comprises: Offline assistance tools; Allocating the data to be annotated to the at least one terminal device to enable the at least one subject to annotate the data to be annotated based on the annotation template and the annotation auxiliary tool specifically includes: Call offline auxiliary tools; Preprocessing the data to be labeled by the offline auxiliary tool to obtain processed data to be labeled, wherein the preprocessing includes: initially labeling the data to be labeled and generating at least one of labeling prompt information for the data to be labeled; The processed data to be labeled are distributed to a terminal device used by the at least one subject, so that the at least one subject performs the labeling task based on the labeling template and the processed data to be labeled.

8. The method of claim 6, wherein a labeled agent is deployed in the artificial intelligence model; Sending the marking template and the loading link of the online auxiliary tool to a terminal device used by the at least one subject so that the terminal device acquires the marking template, and loads the online auxiliary tool based on the loading link, and the at least one subject performs the marking task based on the marking template and the online auxiliary tool, specifically includes: Sending the annotation template and the loading link of the online auxiliary tool to the terminal device used by the at least one subject, and inputting the query information sent by the at least one subject through the online auxiliary tool during the annotation task execution process into the annotation agent to obtain the reply information output by the annotation agent; The reply information is sent to a terminal device used by the at least one subject, so that the at least one subject performs the marking task based on the reply information and the marking template.

9. The method according to claim 1, before determining at least one subject for performing the labeling task from among the candidate subjects according to the task portrait and the acquired subject attribute information of each candidate subject, the method further comprises: Inputting the data to be labeled into the artificial intelligence model to determine the data portrait corresponding to the data to be labeled through the artificial intelligence model; According to the data portrait, determining whether the data to be labeled meets a preset labeling condition; Determining at least one subject for performing the labeling task from among the candidate subjects according to the task portrait and the acquired subject attribute information of each candidate subject, specifically includes: When it is determined that the data to be annotated meets the annotation condition, at least one subject for executing the annotation task is determined from the candidate subjects according to the task portrait and the acquired subject attribute information of each candidate subject.

10. The method of claim 9, further comprising: When it is determined that the data to be annotated does not meet the annotation condition, a prompt message is sent to the user to prompt the user to resend the data to be annotated.

11. The method according to any one of claims 1 to 10, wherein a distribution agent is deployed in the artificial intelligence model; Determining, by means of the artificial intelligence model, a task profile for describing the labeling task corresponding to the labeling request, and determining, according to the task profile and the acquired subject attribute information of each candidate subject, at least one subject for performing the labeling task from among the candidate subjects, specifically includes: Through the assigned intelligent agent deployed in the artificial intelligence model, a task portrait for describing the labeling task corresponding to the labeling request is determined, and based on the task portrait and the acquired subject attribute information of each candidate subject, at least one subject for performing the labeling task is determined from the candidate subjects.

12. The method of claim 1, further comprising: Acquire task evaluation information, where the task evaluation information is generated when the at least one subject performs the labeling task, and the task evaluation information includes: at least one of a task log and behavior data, where the behavior data is monitored through a preset tracking point when the at least one subject performs the labeling task; Inputting the task evaluation information into the artificial intelligence model to determine, through the artificial intelligence model, whether there is a problem when the at least one subject performs the labeling task, and generating warning information when it is determined that there is a problem when the at least one subject performs the labeling task; A task warning is performed according to the warning information.

13. The method of claim 12, wherein the task evaluation information is input into the artificial intelligence model to determine, through the artificial intelligence model, whether there are problems when the at least one subject performs the labeling task, specifically comprising: Inputting the task evaluation information and the quality inspection scope information into the artificial intelligence model, so that the artificial intelligence model determines the object to be inspected from the objects involved when the at least one subject performs the labeling task according to the quality inspection scope information, and determines whether there is a problem with the object to be inspected when the at least one subject performs the labeling task according to the task evaluation information; When determining that the at least one subject has a problem in performing the labeling task, generating warning information specifically includes: When it is determined that there is a problem with the object to be inspected when the at least one subject performs the labeling task, early warning information for the object to be inspected is generated.

14. The method according to claim 13, determining whether there is a problem with the object to be inspected when the at least one subject performs the labeling task according to the task evaluation information, specifically comprising: Determine a quality inspection tool for the object to be inspected; The quality inspection tool is called to determine, based on the task evaluation information, whether there is a problem with the object to be quality inspected when the at least one subject performs the labeling task.

15. The method according to any one of claims 12 to 14, wherein a quality inspection agent is deployed in the artificial intelligence model; Inputting the task evaluation information into the artificial intelligence model to determine, through the artificial intelligence model, whether there is a problem when the at least one subject performs the labeling task, and generating early warning information when it is determined that there is a problem when the at least one subject performs the labeling task, specifically includes: The task evaluation information is input into the artificial intelligence model to determine whether there are problems when at least one subject performs the labeling task through the quality inspection agent deployed in the artificial intelligence model, and generate early warning information when it is determined that there are problems when at least one subject performs the labeling task.

16. A labeling task execution device, wherein an artificial intelligence model is deployed in the device, comprising: A first acquisition module is used to obtain a user's annotation request; A determination module, used to determine the user's data to be annotated and annotation requirement information according to the annotation request, wherein the annotation requirement information is used to indicate the user's annotation requirement for the data to be annotated; An input module, used to input the labeling requirement information into the deployed artificial intelligence model, so as to determine, through the artificial intelligence model, a task profile for describing the labeling task corresponding to the labeling request, and determine at least one subject for performing the labeling task from among the candidate subjects according to the task profile and the subject attribute information of each candidate subject obtained; The allocation module is used to allocate the data to be labeled to the terminal device used by the at least one subject to perform the labeling task.

17. An electronic device comprising: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method as claimed in any one of claims 1 to 15 by running the executable instructions.

18. A computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the method as claimed in any one of claims 1 to 15.

19. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 15.

Images