Method and device for allocating training data labeling tasks, and computing equipment

Abstract

A method and device for allocating training data labeling tasks, a storage medium and a computing device, the method comprising: receiving task creation information, the task creation information indicating a plurality of training data labeling tasks and the type of each training data labeling task; for each type of training data labeling task, identifying a task execution terminal that matches the type; and allocating each type of training data labeling task to a task execution terminal that matches the type. Through the solution of the present invention, a large number of training data labeling tasks can be automatically and efficiently allocated to each task execution terminal, thereby improving the efficiency of allocation.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for allocating training data labeling tasks, a storage medium, and a computing device. Background Art

[0002] Training data annotation refers to the process of describing or marking training data such as text, images, and voice (for example, marking the positions of the left and right outer corners of the eye on a face sample image) so that the annotated training data can be used for machine learning. With the development of technology in fields such as artificial intelligence, the demand for training data annotation by data annotation demanders (such as technology companies in fields such as artificial intelligence) is increasing. Data annotation demanders will send a large amount of training data that needs to be annotated to the data annotation platform. For the training data sent by the data annotation demanders, the data annotation platform needs to create a large number of training data annotation tasks, and the task execution terminals of multiple data annotation executors (for example, execution entities specializing in data annotation) will annotate the training data.

[0003] Therefore, there is an urgent need for a method for allocating training data labeling tasks that can efficiently allocate a large number of training data labeling tasks, thereby further improving the efficiency of completing training data labeling tasks. Summary of the invention

[0004] The technical problem solved by the present invention is to provide a method for allocating training data labeling tasks, which can efficiently allocate a large number of training data labeling tasks, thereby further improving the efficiency of completing the training data labeling tasks.

[0005] To solve the above technical problems, an embodiment of the present invention provides a method for allocating training data labeling tasks, the method comprising: receiving task creation information, the task creation information indicating multiple training data labeling tasks and the type of each training data labeling task; for each type of training data labeling task, identifying a task execution terminal that matches the type; and allocating each type of training data labeling task to a task execution terminal that matches it.

[0006] Optionally, assigning each type of training data labeling task to a task execution terminal that matches it includes: for each type of training data labeling task, dividing the training data labeling task of this type according to the number of task execution terminals that match this type, so as to obtain the training data labeling tasks that need to be completed by each task execution terminal; for each task execution terminal, adding the training data labeling task that it needs to complete to its task list.

[0007] Optionally, assigning each type of training data labeling task to a task execution terminal that matches it also includes: monitoring status information of each task execution terminal, and when its status information indicates that it is in an idle state, selecting the next training data labeling task from its task list; and sending the training data corresponding to the next training data labeling task to the task execution terminal.

[0008] Optionally, the task list includes first progress information, and the first progress information is the ratio of the number of completed training data labeling tasks to the number of training data labeling tasks that need to be completed. The method also includes: monitoring the first progress information of each task execution terminal; when the first progress information of any task execution terminal reaches a first preset threshold, determining whether it is necessary to reallocate the tasks to be completed, wherein the tasks to be completed are training data labeling tasks that are not completed by terminals of the first type, the terminals of the first type are all task execution terminals of the same type as those matching the first terminal, and the first terminal is a task execution terminal whose first progress information reaches the first preset threshold; if so, reallocating the tasks to be completed to the first type of terminals.

[0009] Optionally, determining whether the to-be-completed tasks need to be reallocated includes: counting the number of the to-be-completed tasks, and calculating the mean of the number of the to-be-completed tasks based on the number of the first type of terminals; comparing the number of unfinished training data labeling tasks of the first terminal with the mean, and if the number of unfinished training data labeling tasks of the first terminal and the mean meet preset conditions, determining that the to-be-completed tasks need to be reallocated.

[0010] Optionally, the task list also includes second progress information, and reallocating the to-be-completed tasks to the first type of terminal includes: reading the second progress information of the first type of terminal, wherein the second progress information is the number of training data labeling tasks completed within a preset time; and dividing the to-be-completed tasks according to the second progress information of the first type of terminal to update the training data labeling that needs to be completed corresponding to the first type of terminal.

[0011] Optionally, the task execution terminal has a type label, and for each type of training data labeling task, identifying the task execution terminal that matches the type includes: for each type of training data labeling task, searching for a task execution terminal with the same type label and / or a blank label as the type to obtain a task execution terminal that matches the type.

[0012] Optionally, before dividing the to-be-completed tasks according to the second progress information of the first type of terminals, it also includes: if the type tag of at least one task execution terminal among the first type of terminals is a blank tag, and the second progress information of the at least one task execution terminal reaches a second preset threshold, then the type tag of the at least one task execution terminal is modified to a type tag that is the same as the first type, wherein the first type is a type that matches the first terminal.

[0013] Optionally, if the type label of at least one task execution terminal among the first type of terminals is not a blank label, and the second progress information of the at least one task execution terminal is less than the second preset threshold, the type label of the at least one task execution terminal is eliminated, wherein the first type is the type matched by the first terminal. In order to solve the above technical problems, an embodiment of the present invention also proposes a device for allocating training data labeling tasks, the device comprising: a receiving module for receiving task creation information, the task creation information indicating multiple training data labeling tasks and the type of each training data labeling task; an identification module for identifying, for each type of training data labeling task, a task execution terminal that matches the type; and an allocation module for allocating each type of training data labeling task to a task execution terminal that matches it.

[0014] An embodiment of the present invention further provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned method for allocating training data labeling tasks are executed.

[0015] An embodiment of the present invention further proposes a computing device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor executes the steps of the above-mentioned method for allocating training data labeling tasks when running the computer program.

[0016] Compared with the prior art, the technical solution of the embodiment of the present invention has the following beneficial effects: the embodiment of the present invention provides a method for allocating training data labeling tasks, the method comprising: receiving task creation information, the task creation information indicating multiple training data labeling tasks and the type of each training data labeling task; for each type of training data labeling task, identifying a task execution terminal that matches the type; and allocating each type of training data labeling task to a task execution terminal that matches it. In the embodiment of the present invention, the received task creation information includes the types of each training data labeling task, and the data labeling platform can search for matching task execution terminals according to the types of the training data labeling tasks. When allocating training data labeling tasks, each training data labeling task can be allocated to a task execution terminal that matches the type of the task, so that a large number of training data labeling tasks can be efficiently allocated to appropriate task execution terminals to improve labeling efficiency.

[0017] Furthermore, in the embodiment of the present invention, the data annotation platform can also monitor the first progress information of each task execution terminal. When the first progress information of any task execution terminal reaches the first preset threshold, that is, when the completion progress of any task execution terminal is faster, the progress of the task execution terminal is further compared with the average progress of all task execution terminals of the same type. If the two meet the preset conditions, that is, the difference between the two is large, the unfinished tasks of all the above task execution terminals are reallocated to the matching task execution terminals. Therefore, the scheme of the embodiment of the present invention can dynamically adjust the number of tasks assigned to each task execution terminal according to the completion progress, so that the allocation of training data annotation tasks is more reasonable, and the annotation efficiency can be further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a schematic diagram of an application scenario of a method for allocating training data labeling tasks in an embodiment of the present invention.

[0019] Figure 2 It is a flowchart of a method for allocating training data labeling tasks in an embodiment of the present invention.

[0020] Figure 3 It is a structural schematic diagram of a device for allocating training data labeling tasks in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] As described in the background technology, there is an urgent need for a method for allocating training data labeling tasks, which can efficiently allocate a large number of training data labeling tasks, thereby further improving the efficiency of completing the training data labeling tasks.

[0022] The inventor of the present invention has found through research that in the prior art, the administrator of the data annotation platform usually determines the annotators for each training data annotation task in turn, and each annotator of the data annotation executor needs to find the training data annotation task assigned by the administrator among a large number of training data annotation tasks, and perform annotation after receiving the task on the data annotation platform. Since the number of annotators and the number of training data annotation tasks are relatively large, it will take a lot of time for the administrator to assign training data annotation tasks; in addition, when annotators receive training data annotation tasks, they also need to find the tasks assigned by the administrator under a large number of training data annotation tasks, which may easily lead to the receipt of the wrong tasks due to similar names of training data annotation tasks, resulting in the postponement of urgent training data annotation tasks, etc. Therefore, the allocation process of this method is cumbersome and complicated, and heavily relies on manual operations.

[0023] In order to solve the above technical problems, an embodiment of the present invention provides a method for assigning training data labeling tasks, the method comprising: receiving task creation information, the task creation information indicating multiple training data labeling tasks and the type of each training data labeling task; for each type of training data labeling task, identifying a task execution terminal that matches the type; and assigning each type of training data labeling task to a task execution terminal that matches it. In an embodiment of the present invention, the received task creation information includes the types of each training data labeling task, and the data labeling platform can search for matching task execution terminals according to the types of the training data labeling tasks. When assigning training data labeling tasks, each training data labeling task can be assigned to a task execution terminal that matches the type of the task, so that a large number of training data labeling tasks can be efficiently assigned to appropriate task execution terminals to improve labeling efficiency.

[0024] In order to make the above-mentioned objects, features and beneficial effects of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0025] refer to Figure 1 , Figure 1 1 is a schematic diagram of an application scenario of a method for allocating training data annotation tasks in an embodiment of the present invention. The method can be executed by a data annotation platform 11, which can be a platform of a data annotation demander (not shown) itself, and the data annotation demander can be a technology company in the field of artificial intelligence, but is not limited thereto. The data annotation platform 11 can also be a third-party platform independent of the data annotation demander and multiple data annotation executors (for example, an executor specializing in data annotation), but is not limited thereto.

[0026] Furthermore, the data annotation platform 11 may include at least one server (not shown), and the data annotation platform 11 is coupled to a database server (not shown), and the database server is used to store training data, which may be an internal server of the data annotation demander, or other servers storing training data, but is not limited thereto. The data annotation platform 11 may obtain the training data that needs to be annotated from the database server, and create multiple training data annotation tasks for the training data that needs to be annotated, and then assign the multiple training data annotation tasks to the task execution terminals 12 of multiple data annotation executors for annotation. When annotating the training data, the training data may be automatically annotated by multiple task execution terminals 12, or the training data may be annotated manually by the task execution terminals 12, and the task execution terminals 12 may be terminals such as mobile phones, computers, and tablet computers.

[0027] Figure 2 is a flow chart of a method for allocating training data labeling tasks in an embodiment of the present invention. Figure 2 The method for assigning training data labeling tasks shown may include the following steps:

[0028] Step S201: receiving task creation information, wherein the task creation information indicates a plurality of training data labeling tasks and a type of each training data labeling task;

[0029] Step S202: for each type of training data labeling task, identifying a task execution terminal matching the type;

[0030] Step S203: Allocate each type of training data labeling task to a task execution terminal that matches it.

[0031] In the specific implementation of step S201, the task creation information received by the data annotation platform may be sent by the data annotation demander, and the task creation information may indicate multiple training data annotation tasks and the type of each training data annotation task. The type of the training data annotation task is determined by the type of training data corresponding to the task, which may be a picture, text, video, etc. For example, if the training data is a picture, then the type of the corresponding training data annotation task is a picture. The multiple training data annotation tasks may include at least one type of training data annotation task. For example, the training data that the data annotation demander needs to annotate may include both picture data and text data.

[0032] Specifically, when the data annotation demander needs to annotate the training data, the task creation information is sent to the data annotation platform. The task creation information may include the training data to be annotated and the type of training data (text, picture, voice, etc.); the task creation information may also include the storage address of the training data to be annotated, the type of training data and the number of each type of training data. The data annotation platform obtains the training data from the database server according to the type and number of training data, but is not limited to this. In a non-limiting embodiment of the present invention, the task creation information may also include at least one task execution terminal specified by the data annotation demander.

[0033] Furthermore, the data annotation platform creates multiple training data annotation tasks according to the received task creation information, and each training data annotation task may include a first preset number of training data of the same type, in which case the type of the first preset number of training data is the type of the training data annotation task. The first preset number may be predetermined by the data annotation platform or may be specified by the data annotation demander, for example, the first preset number may be included in the task creation information.

[0034] In the specific implementation of step S202, each task execution terminal may have at least one type tag, and the data annotation platform has a terminal list, which may include the terminal identifiers and type tags of each task execution terminal of the data annotation executor, and the terminal identifier corresponds to the task execution terminal one by one, that is, the terminal identifier can uniquely determine the task execution terminal. The type tag may indicate the type of data that the task execution terminal is good at annotating. For example, if the type tag of task execution terminal A in the terminal list is "picture", it means that task execution terminal A is good at annotating picture data, that is, task execution terminal A has higher accuracy and / or higher efficiency when annotating picture data. The type tag may also be a blank tag, which is used to indicate that the type of data that the task execution terminal is good at annotating is still uncertain. For example, if the type tag of task execution terminal B is not included in the terminal list, it is considered that the type tag of task execution terminal B is a blank tag.

[0035] It should be noted that when a task execution terminal has multiple type labels, the task execution terminal can match multiple types of training data labeling tasks.

[0036] Furthermore, for each type of training data annotation task, the data annotation platform can search for a task execution terminal with a type label that is the same as the type in the terminal list according to the type, that is, the task execution terminal with the same type label as the type is used as the task execution terminal matching the type. For example, if the type of the training data annotation task is picture, the data annotation platform can search for a task execution terminal with a type label of picture in the terminal list.

[0037] In a non-limiting embodiment of the present invention, a task execution terminal with a blank label may also be regarded as a task execution terminal matching the type, and the task execution terminal matching the type includes a task execution terminal having the same type label and / or a blank label as the type.

[0038] It should be clear that if the task creation information includes at least one task execution terminal specified by the data annotation demander, then the at least one task execution terminal specified by the data annotation demander can also be used as a matching task execution terminal.

[0039] In the specific implementation of step S203, each task execution terminal has a corresponding task list, and the task list may include the task number and status information of the training data labeling task, but is not limited thereto. The task number can be used to uniquely identify the training data labeling task. The status information can be used to indicate whether the task execution terminal has completed the training data labeling task. For each training data labeling task, if the task execution terminal has completed labeling all the training data corresponding to the task, the status information of the training data labeling task is "completed", otherwise, it is "uncompleted", and the sum of the number of unfinished training data labeling tasks and the number of completed training data labeling tasks in the task list is the number of training data labeling tasks that the task execution terminal needs to complete.

[0040] Furthermore, for each type of training data labeling task, after the data labeling platform matches the corresponding task execution terminal, it can determine the training data labeling tasks that each task execution terminal needs to complete, wherein the task assigned to each task execution terminal by the data labeling platform is the training data labeling task that the terminal needs to complete.

[0041] Specifically, for each type of training data labeling task, the data labeling platform can divide the training data labeling task of this type into multiple parts according to the number of task execution terminals that match this type, so as to obtain the training data labeling tasks that need to be completed by each task execution terminal; for each task execution terminal, the task number of the training data labeling task that needs to be completed by the terminal is written into its task list.

[0042] In a non-limiting embodiment of the present invention, for each type of training data labeling task, the training data labeling task of that type can be evenly distributed to each task execution terminal that matches the type. For example, if there are M training data labeling tasks of the image type and there are N matching task execution terminals, then the M training data labeling tasks are evenly distributed to the N task execution terminals, that is, each task execution terminal needs to complete M / N training data labeling tasks of the image type, and the task numbers of the M / N training data labeling tasks are written into the task lists of the N task execution terminals respectively.

[0043] Furthermore, the terminal list may also include the labeling speed information of each task execution terminal, and the labeling speed information may be used to indicate the speed at which the task execution terminal labels the training data. For example, the labeling speed information may be the time taken by the task execution terminal to complete a second preset number of training data labeling tasks, and the second preset number may be predetermined by the data labeling platform. Thus, the data labeling platform may also be divided according to the labeling speed information of each task execution terminal, that is, the task execution terminal with a faster labeling speed is assigned more training data labeling tasks, thereby making the distribution of training data labeling tasks more reasonable and improving the labeling efficiency.

[0044] Furthermore, for each task execution terminal, after the data annotation platform adds the training data annotation task that needs to be completed to its task list, it can monitor the status information of the task execution terminal. When the task execution terminal is annotating the training data, its status information can be "busy green"; when the task execution terminal completes any training data annotation task and has not started the next training data annotation task, its status information can be "idle". When the data annotation platform monitors the status information indicating that the task execution terminal is idle, it can select any unfinished training data annotation task from the task execution terminal's task list as the next training data annotation task; and send the training data corresponding to the next training data annotation task to the task execution terminal for annotation.

[0045] Furthermore, the task list may also include first progress information, where the first progress information is the ratio of the number of training data labeling tasks completed by the task execution terminal to the number of tasks that need to be completed. During the process of the task execution terminal labeling the training data, the data labeling platform may also monitor the first progress information of each task execution terminal. For example, the data labeling platform may periodically read the first progress information of each task execution terminal.

[0046] In a non-limiting embodiment of the present invention, each task execution terminal has a unique type tag, that is, the types of the training data annotation tasks in the task list are the same. In this scenario, if the first progress information of any task execution terminal is monitored to reach the first preset threshold, the data annotation platform records the task execution terminal whose first progress information reaches the first preset threshold as the first terminal, and records the type of the training data annotation task matched by the first terminal as the first type. It should be noted that the first preset threshold can be predetermined or determined by the data annotation platform in real time. For example, it can be determined based on the annotation speed information of each task execution terminal.

[0047] It should be clear that, as described above, the task execution terminal may have at least one type tag. When the task execution terminal has multiple type tags and is assigned to multiple types of training data labeling tasks, the task list of the terminal includes multiple types of training data labeling tasks, and the task list may include multiple first progress information, and the first progress information corresponds to the type of training data labeling task. For example, the task list of the task execution terminal includes image-type training data labeling tasks and text-type training data labeling tasks, and the task list includes the first progress information of the image type and the first progress information of the text type.

[0048] It should be clarified that, if the task list of the first terminal includes a plurality of first progress information, the first type is the type corresponding to the first progress information that reaches the first preset threshold.

[0049] Furthermore, the data annotation platform can determine whether it is necessary to reallocate the tasks to be completed. Specifically, the data annotation platform reads the task list of each task execution terminal in the first type of terminal to determine the unfinished training data annotation tasks of the first type of terminal, which are recorded as unfinished tasks, wherein the first type of terminal refers to all task execution terminals matching the first type (including the first terminal). For example, if the first type is a picture, then all unfinished training data annotation tasks of the task execution terminals matching the picture type are read.

[0050] Further, according to the number of first-type terminals and the number of tasks to be completed, the average of the number of tasks to be completed is calculated, where the average refers to the result of dividing the number of tasks to be completed by the number of first-type terminals. If the average and the number of uncompleted first-type tasks of the first terminal meet a preset condition, it can be determined that the tasks to be completed need to be reallocated.

[0051] As a non-limiting embodiment, the preset condition may be that the difference between the number of uncompleted tasks of the first type by the first terminal minus the mean is greater than a preset difference, or the number of uncompleted tasks of the first type by the first terminal exceeds a preset multiple of the mean (for example, more than twice the mean), but is not limited thereto. The preset difference and the preset multiple may be predetermined by the data annotation platform, or may be determined by the data annotation platform through real-time calculation.

[0052] Furthermore, if it is necessary to reallocate the tasks to be completed, the data annotation platform may reallocate the tasks to be completed to the first type of terminals.

[0053] Furthermore, the data labeling platform can also adjust the first type of terminal before reallocating the tasks to be completed. Specifically, the task list can also include second progress information, which can be used to indicate the speed at which the task execution terminal completes the training data labeling task, for example: the second progress information is the number of training data labeling tasks completed by the task execution terminal within a preset time. If the second progress information of any task execution terminal in the first type of terminal is less than the second preset threshold, it is judged that the task execution terminal completes the first type of training data labeling task slowly and is not suitable for labeling the first type of training data. Therefore, the matching relationship between the task execution terminal and the first type can be cancelled, that is, the task execution terminal is removed from the first type of terminal, thereby adjusting the first type of terminal. The preset time can be preset by the data labeling platform, for example, it can be 1 hour. The second preset threshold can be predetermined by the data labeling platform, or it can be determined by the data labeling platform in real time. The second progress information can be updated regularly.

[0054] Furthermore, if the type label of any task execution terminal in the first type of terminals is not a blank label, and the second progress information of the task execution terminal is less than a second preset threshold, the data annotation platform can delete the type label of the terminal that is the same as the first type in the terminal list, and in the subsequent reallocation process, the to-be-completed task will no longer be assigned to the task execution terminal.

[0055] It should be noted that if the task list includes multiple types of training data labeling tasks, the task list may include multiple second progress information, and the second progress information corresponds to the type of training data labeling task. The second progress information described above is the second progress information corresponding to the first type.

[0056] Furthermore, if the type label of any task execution terminal in the first type of terminals is a blank label, and the second progress information of the task execution terminal reaches a second preset threshold, the type label of the terminal in the terminal list can be modified to a type label that is the same as the first type, thereby determining the type of training data that the task execution terminal is good at labeling, and in subsequent task allocation, it can be avoided to assign training data labeling tasks of mismatched types to the terminal.

[0057] Furthermore, after the data annotation platform adjusts the first type of terminal, it reallocates the to-be-completed tasks to the adjusted first type of terminal. The data annotation platform can evenly distribute the to-be-completed tasks to the first type of terminal, or divide the to-be-completed tasks according to the second progress information of the first type of terminal to obtain the to-be-completed tasks assigned to each terminal in the first type of terminal. The more first type of tasks completed within the preset time, the more to-be-completed tasks assigned. For each terminal in the first type of terminal, the task number of the unfinished first type of training data annotation task can be deleted from the task list, and the task number of the latest assigned to-be-completed task can be written in to update the training data annotation tasks that need to be completed by the first type of terminal.

[0058] refer to Figure 3 , Figure 3 3 is a schematic diagram of a structure of a training data labeling task distribution device in an embodiment of the present invention. The training data labeling task distribution device may include: a receiving module 31, a recognition module 32, and a distribution module 33.

[0059] Among them, the receiving module 31 is used to receive task creation information, which indicates multiple training data labeling tasks and the type of each training data labeling task; the identification module 32 is used to identify the task execution terminal that matches each type of training data labeling task; the allocation module 33 is used to allocate each type of training data labeling task to the task execution terminal that matches it.

[0060] For more information about the working principle and working method of the training data labeling task allocation device, please refer to the above Figure 1 and Figure 2 The relevant description is not repeated here.

[0061] The embodiment of the present invention further provides a storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method for assigning the training data labeling task are executed. The storage medium may include ROM, RAM, a magnetic disk or an optical disk, etc. The storage medium may also include a non-volatile memory or a non-transitory memory, etc.

[0062] An embodiment of the present invention further provides a computing device, the computing device comprising a memory and a processor, the memory storing a computer program that can be run on the processor, and the processor executing the steps of the method for the training data labeling task when running the computer program. The computing device may be a server, etc.

[0063] It should be understood that the term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article indicates that the associated objects before and after are in an "or" relationship.

[0064] The "plurality" appearing in the embodiments of the present application refers to two or more.

[0065] The first, second, etc. descriptions appearing in the embodiments of the present application are only used for illustration and distinction of the description objects. There is no order, nor do they indicate any special limitation on the number of devices in the embodiments of the present application, and cannot constitute any limitation on the embodiments of the present application.

[0066] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the scope defined by the claims.

Claims

1. A method for allocating training data labeling tasks, characterized in that: The method comprises: receiving task creation information, wherein the task creation information indicates a plurality of training data labeling tasks and a type of each training data labeling task; For each type of training data labeling task, identify the task execution terminal that matches the type; Allocating each type of training data labeling task to a task execution terminal that matches the task execution terminal, including: for each type of training data labeling task, dividing the type of training data labeling task according to the number of task execution terminals that match the type, so as to obtain training data labeling tasks that need to be completed by each task execution terminal; for each task execution terminal, adding the training data labeling task that needs to be completed to its task list; The task list includes first progress information, where the first progress information is the ratio of the number of completed training data labeling tasks to the number of training data labeling tasks that need to be completed. The method further includes: monitoring the first progress information of each task execution terminal; when the first progress information of any task execution terminal reaches a first preset threshold, determining whether it is necessary to reallocate the unfinished task, wherein the unfinished task is an unfinished training data labeling task of a first type terminal, the first type terminal is all task execution terminals of the same type as that matched with the first terminal, and the first terminal is a task execution terminal whose first progress information reaches the first preset threshold; if yes, reallocating the unfinished task to the first type terminal; The determining whether it is necessary to reallocate the tasks to be completed includes: counting the number of the tasks to be completed, and calculating the mean of the number of the tasks to be completed based on the number of terminals of the first type; comparing the number of unfinished training data labeling tasks of the first terminal with the mean, and if the number of unfinished training data labeling tasks of the first terminal and the mean meet a preset condition, determining that it is necessary to reallocate the tasks to be completed.

2. The method for assigning training data labeling tasks according to claim 1, characterized in that: Assigning each type of training data labeling task to the task execution terminal that matches it also includes: Monitor the status information of each task execution terminal, and when its status information indicates that it is in an idle state, select the next training data labeling task from its task list; The training data corresponding to the next training data labeling task is sent to the task execution terminal.

3. The method for assigning training data labeling tasks according to claim 1, characterized in that: The task list further includes second progress information, and reallocating the to-be-completed task to the first type of terminal includes: Reading second progress information of the first type of terminal, wherein the second progress information is the number of training data labeling tasks completed within a preset time; The to-be-completed tasks are divided according to the second progress information of the first type of terminals to update the training data labeling tasks that need to be completed corresponding to the first type of terminals.

4. The method for assigning training data labeling tasks according to claim 3, characterized in that: The task execution terminal has a type label. For each type of training data labeling task, identifying the task execution terminal matching the type includes: For each type of training data labeling task, find a task execution terminal with the same type label and / or blank label as that type to obtain a task execution terminal matching that type.

5. The method for assigning training data labeling tasks according to claim 4, characterized in that: Before dividing the to-be-completed tasks according to the second progress information of the first type of terminal, the method further includes: If the type tag of at least one task execution terminal among the first type of terminals is a blank tag, and the second progress information of the at least one task execution terminal reaches a second preset threshold, the type tag of the at least one task execution terminal is modified to a type tag that is the same as the first type, wherein the first type is the type that matches the first terminal.

6. The method for assigning training data labeling tasks according to claim 5, characterized in that: Before dividing the to-be-completed tasks according to the second progress information of the first type of terminal, the method further includes: If the type tag of at least one task execution terminal among the first type terminals is not a blank tag, and the second progress information of the at least one task execution terminal is less than the second preset threshold, the type tag of the at least one task execution terminal is removed.

7. A device for allocating training data labeling tasks, characterized in that: The device is based on the method for allocating training data labeling tasks according to any one of claims 1 to 6, comprising: A receiving module, configured to receive task creation information, wherein the task creation information indicates a plurality of training data labeling tasks and a type of each training data labeling task; An identification module, used for identifying, for each type of training data labeling task, a task execution terminal matching the type; The allocation module is used to allocate each type of training data labeling task to the task execution terminal that matches it.

8. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for assigning training data labeling tasks according to any one of claims 1 to 6 are performed.

9. A computing device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor runs the computer program, the processor performs the steps of the method for allocating training data labeling tasks according to any one of claims 1 to 6.

Images