A method, device, equipment and medium for processing perception data annotation tasks

Abstract

The present invention discloses a processing method, device, equipment and medium for perception data labeling tasks, which relates to the field of autonomous driving, and the method includes: obtaining learning parameters of personnel to be operated and obtaining information to be delivered of items to be delivered; inputting the learning parameters and the information to be delivered into a task splitting model to obtain labeling parameters output by the task splitting model; based on the labeling parameters, performing task allocation for items to be delivered; wherein, the higher the value of the labeling parameter corresponding to the personnel to be operated, the higher the priority of the personnel to be operated in the processing of perception data labeling tasks. The present invention facilitates batch inspection of operator quality, saves the time cost of inspectors, and avoids the problem of misidentification during manual inspection, and provides a universal task splitting solution for data labeling in autonomous driving scenarios, greatly shortening the time from the beginning of labeling personnel's learning to the final start of formal projects.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method, device, equipment and medium for processing a perception data labeling task. Background Art

[0002] With the rapid development of the field of autonomous driving, the requirements for training data of various neural network models required for autonomous driving scenarios are also increasing. Data labeling can help these models improve recognition accuracy and precision. Correspondingly, the requirements for data labeling of these models are becoming increasingly complex, and the requirements for data labeling accuracy are becoming increasingly stringent. However, at present, labelers need strong professional knowledge to perform the assigned data labeling tasks, and it is difficult for newcomers to get started. There is currently no effective way to reduce the learning cost of newcomers. When newcomers get started with new labeling tasks, the average learning time is 1-2 weeks, which not only results in high data labeling costs, but also leads to low processing efficiency of the entire new project.

[0003] Therefore, it is necessary to provide a processing method for the autonomous driving data labeling task to improve the accuracy of the labeling. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a method, device, equipment and medium for processing perception data labeling tasks, so as to solve the problems of high data labeling cost and low processing efficiency in the field of autonomous driving.

[0005] According to a first aspect, an embodiment of the present invention provides a method for processing a perception data labeling task, the method comprising:

[0006] Obtain the learning parameters of each person to be operated and the to-be-delivered information of the to-be-delivered project; the to-be-delivered information includes all subtasks included in the to-be-delivered project, the interrelated task types in the subtasks, the basic task information of each subtask, the task processing flow, the task node and the task delivery cycle, and, when it is determined that the granularity of the subtask is less than the preset value, merge the subtasks with a granularity less than the preset value according to the interrelated task types in the subtasks, the basic task information of each subtask, the task processing flow, the task node and the task delivery cycle; the learning parameters include the actual operation time of the person to be operated to process each historical annotation task and the number of feedbacks for each inspection node of the historical annotation task, the number of feedbacks is the rounds used when the current node passes, the actual operation time is the sum of the actual operation time corresponding to each round when the historical annotation task is completed, and the inspection nodes include the audit nodes, quality inspection nodes and acceptance nodes that are performed sequentially;

[0007] The learning parameters and the information to be delivered are input into the trained task splitting model to obtain the labeling parameters output by the task splitting model; the labeling parameters include the task processing capability evaluation parameters of the personnel to be operated to handle the items to be delivered, and the operating capacity required for the labeling job formed after each subtask of the items to be delivered is split; the task processing capability evaluation parameters include the capability evaluation value, timeliness evaluation value and accuracy evaluation value of the personnel to be operated; the timeliness evaluation value is obtained based on the timeliness value, the accuracy evaluation value is obtained based on the accuracy value, the timeliness value forms a mapping relationship with the actual operation time, and the accuracy value forms a mapping relationship with the number of feedbacks of each inspection node of the historical labeling task; the task splitting model is obtained based on the actual operation time and the number of feedbacks of each inspection node obtained by the historical operators in handling the historical labeling tasks, and the task splitting template formed based on the task flow scheduling engine is trained;

[0008] Based on the annotation parameters, task allocation is performed for the project to be delivered; the higher the value of the annotation parameter corresponding to the operator to be operated is, the higher the priority of the operator to be operated in processing the perception data annotation task is.

[0009] In combination with the first aspect, in a first implementation of the first aspect, the learning parameter is obtained by the following steps:

[0010] Collect historical annotation tasks and historical annotation data of historical annotation tasks;

[0011] Form standard evaluation parameters based on the historical node record data of the historical annotation task;

[0012] According to the labeling evaluation parameters and the first processing situation recorded when each operator handles the historical labeling data of the historical labeling task of the first preset number of cycles, the processing ability of the operator is evaluated to obtain the learning parameters of each operator; the historical labeling data is the inspection data recorded when the historical labeling task passes through each inspection node.

[0013] In combination with the first aspect, in a second implementation of the first aspect, the allocating tasks of the to-be-delivered project based on the annotation parameters specifically includes:

[0014] Determine the job priority of each person waiting to be operated according to the first parameter weight of each sub-parameter in the task processing capability evaluation parameter;

[0015] Based on the job capacity of the labeling job, the subtask capacity of the subtask consisting of at least one labeling job, and the job priority of the person to be delivered, the items to be delivered are dispatched to the person to be delivered, with the condition that the labeling job and subtasks are completed on time and with quality; the subtask capacity is obtained by accumulating the job capacity of all labeling jobs to which the subtask belongs.

[0016] In combination with the second implementation manner of the first aspect, in the third implementation manner of the first aspect, according to the job capacity of the labeling job, the subtask capacity of the subtask composed of at least one labeling job, and the job priority of the person to be operated, the items to be delivered are dispatched to the person to be delivered on condition that the labeling job and the subtask are completed on time and with quality, specifically including:

[0017] Evaluate the processing quality of the labeling job and the subtasks corresponding to the labeling job according to the information to be delivered, and determine a first probability that the unfinished labeling task will be completed on time and with quality, and a second probability that the unfinished subtask will be completed on time and with quality;

[0018] When it is determined that the first probability and / or the second probability is lower than the preset probability, the personnel to be operated are scheduled according to their job priority, the remaining job capacity of the unfinished marking tasks, and the remaining subtask capacity of the unfinished subtasks.

[0019] In combination with the first aspect, in a fourth implementation of the first aspect, the task flow scheduling engine includes a split setting module, a job initialization module, a job scheduling module, a labeling module, and a quality assurance module;

[0020] The split setting module is used to complete the standardized configuration of the demand elements according to the information to be delivered to the customer, and the standardized configuration is to split different detection objects;

[0021] The job initialization module is used to split the subtasks in the to-be-delivered project based on the standardized settings completed by the split setting module to form a number of labeling jobs;

[0022] The job scheduling module is used to form the scheduling priority among all scheduled jobs based on the preset rule engine, and schedule the labeling jobs;

[0023] The annotation module is used to provide an annotation tool set and maintain independent annotation pages; the tool set is used to provide the annotation work to be completed by the operator to split the annotation work;

[0024] The quality assurance module is used to maintain and create inspection processes for inspection nodes;

[0025] Flow establishment module: used to set up workflow plans, establish forward flow nodes and reverse operation processes between nodes.

[0026] In combination with the fourth implementation of the first aspect, in the fifth implementation of the first aspect, the operation capacity is determined by:

[0027] Obtain the processing status of historical annotation tasks at each checkpoint;

[0028] Perform similarity matching between the items to be delivered and the historical annotation tasks according to the information to be delivered and the historical delivery information of the historical annotation tasks, and obtain a matching result;

[0029] The difficulty of each subtask in the project to be delivered is predicted based on the processing status and matching results, and the job capacity of each labeling job is determined based on the difficulty of the subtask and the splitting of the subtask.

[0030] In combination with the first aspect, in a sixth implementation of the first aspect, the method further includes, before the step of inputting the learning parameters and the information to be delivered into the trained task splitting model to obtain the labeling parameters output by the task splitting model:

[0031] The learning parameters of each worker to be operated are evaluated by a preset first evaluation standard and a preset second parameter weight of each sub-parameter in the preset parameter to determine whether the learning parameter is not lower than the preset parameter;

[0032] When it is determined that the learning parameters are lower than the preset parameters, the historical labeling data of the historical labeling tasks for the second preset number of cycles are handed over to the personnel to be operated. According to the labeling evaluation parameters and the second processing situation recorded when each personnel to be operated processes the historical labeling data of the historical labeling tasks for the second preset number of cycles, the processing capabilities of the personnel to be operated are evaluated, and the learning parameters of each personnel to be operated are updated until the learning parameters of the corresponding personnel to be operated are no lower than the preset parameters.

[0033] According to a second aspect, an embodiment of the present invention further provides a processing device for a perception data labeling task, the device comprising:

[0034] An acquisition module is used to acquire the learning parameters of each person to be operated and the to-be-delivered information of the to-be-delivered project; the to-be-delivered information includes all subtasks included in the to-be-delivered project, the interrelated task types in the subtasks, the basic task information of each subtask, the task processing flow, the task node and the task delivery cycle, and, when it is determined that the granularity of the subtask is less than the preset value, the subtasks with a granularity less than the preset value are merged according to the interrelated task types in the subtasks, the basic task information of each subtask, the task processing flow, the task node and the task delivery cycle; the learning parameters include the actual operation time consumed by the person to be operated to process each historical annotation task and the number of feedbacks for each inspection node of the historical annotation task, the number of feedbacks is the rounds used when the current node passes, the actual operation time is the sum of the actual operation time corresponding to each round when the historical annotation task is completed, and the inspection nodes include the audit nodes, quality inspection nodes and acceptance nodes that are performed sequentially;

[0035] A splitting module is used to input the learning parameters and the information to be delivered into a trained task splitting model to obtain the labeling parameters output by the task splitting model; the labeling parameters include the task processing capability evaluation parameters of the personnel to be operated to handle the items to be delivered, and the operating capacity required for the labeling job formed after each subtask of the items to be delivered is split; the task processing capability evaluation parameters include the capability evaluation value, timeliness evaluation value and accuracy evaluation value of the personnel to be operated; the timeliness evaluation value is obtained based on the timeliness value, the accuracy evaluation value is obtained based on the accuracy value, the timeliness value forms a mapping relationship with the actual operation time, and the accuracy value forms a mapping relationship with the number of feedbacks of each inspection node of the historical labeling task; the task splitting model is trained based on the actual operation time and the number of feedbacks of each inspection node obtained by the historical operators in handling the historical labeling tasks, and the task splitting template formed based on the task flow scheduling engine;

[0036] An allocation module is used to allocate tasks for the projects to be delivered based on the annotation parameters; wherein, the higher the value of the annotation parameter corresponding to the person to be operated is, the higher the priority of the person to be operated is in processing the perception data annotation task.

[0037] According to the third aspect, an embodiment of the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of a processing method for a perception data labeling task as described in any one of the above are implemented.

[0038] According to a fourth aspect, an embodiment of the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a processing method for a perception data labeling task as described in any one of the above.

[0039] The processing method, device, equipment and medium of the perception data labeling task of the present invention obtains the learning parameters of each operator to be operated and the to-be-delivered information of the to-be-delivered project, and inputs these two data into the constructed task splitting model to obtain the labeling parameters output by the task splitting model, which is convenient for batch inspection of the operator's quality, saves the time cost of the inspector, and avoids the problem of misidentification during manual inspection. It provides a universal task splitting solution for data labeling of autonomous driving scenarios, greatly shortens the time for labeling personnel to start learning and finally get started with formal projects. By allocating tasks for the to-be-delivered project through labeling parameters, labeling tasks are preferentially allocated to operators to be operated with high completion quality and fast completion speed, that is, tasks are preferentially allocated to operators to be operated with higher comprehensive capabilities, ensuring that the to-be-delivered projects are delivered to customers on time and in quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The features and advantages of the present invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present invention in any way. In the accompanying drawings:

[0041] Figure 1 A schematic diagram showing a process flow of a method for processing a perception data annotation task provided by the present invention;

[0042] Figure 2 A schematic diagram showing the structure of a method for processing a perception data annotation task provided by the present invention is shown;

[0043] Figure 3 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0045] Autonomous driving technology is rapidly becoming the focus of the automotive industry. With the mature application of end-to-end technologies, the field of autonomous driving has become the main driving force for artificial intelligence innovation, driving artificial intelligence systems to handle complex and unpredictable real-world scenarios. The development of the autonomous driving industry has not only promoted technological innovation, but also brought users a safer and more convenient driving experience.

[0046] Technological innovation is the core driving force for the development of the autonomous driving industry. Breakthroughs in sensor technology, artificial intelligence algorithms, big data processing, and 5G communications will significantly improve the perception, decision-making, and control capabilities of autonomous driving systems. As these technologies continue to mature, autonomous vehicles will achieve a leap from assisted driving to fully unmanned driving, bringing revolutionary changes to transportation. With the rapid development of the field of autonomous driving, the requirements for training data for various neural network models required for autonomous driving scenarios are also increasing. Data labeling can help these models improve recognition accuracy and accuracy. Correspondingly, these models have increasingly complex requirements for data labeling, and the requirements for data labeling accuracy are becoming increasingly stringent. However, at present, labelers need strong professional knowledge to perform the assigned data labeling tasks, and it is difficult for newcomers to get started. There is currently no effective way to reduce the learning cost of newcomers. When newcomers get started with new labeling tasks, the average learning time is 1-2 weeks, which not only causes high data labeling costs, but also leads to low processing efficiency of the entire new project. For example:

[0047] (1) The time spent on admission of new project annotators is relatively long, and their output is relatively low during the admission period, which is a waste of labor efficiency.

[0048] (2) The cost of training inexperienced labelers to become capable labelers is high. This is due to the fact that customer needs are complex and prone to interruption, which leads to high unit costs for the platform to screen and train operators.

[0049] (3) After the batch is submitted for inspection and may be rejected during acceptance, the project delivery is prone to risks.

[0050] Therefore, it is necessary to provide a processing method for the autonomous driving data labeling task to improve the accuracy of the labeling.

[0051] In order to solve the above problems, a method for processing perception data labeling tasks is provided in this embodiment, aiming to better reduce the operating costs of new labeling personnel, thereby improving the accuracy of data labeling. The method for processing perception data labeling tasks in the embodiment of the present invention can be used in electronic devices, including but not limited to computers, mobile terminals, etc. Figure 1FIG. 1 is a flow chart of a method for processing a perception data labeling task according to an embodiment of the present invention. Figure 1 As shown, the method may include the following steps:

[0052] S10, obtain the learning parameters of each person to be operated and obtain the information to be delivered of the project to be delivered. In this method, the information to be delivered includes all subtasks contained in the project to be delivered, the task types related to each other in the subtasks, the basic task information of each subtask, the task processing flow, the task node and the task delivery cycle. The learning parameters include the actual operation time of the person to be operated to process each historical annotation task and the number of feedbacks of each check node of the historical annotation task. The number of feedbacks is the rounds used when the current node passes. The actual operation time is the sum of the actual operation time corresponding to each round when the historical annotation task is completed. The check node includes the audit node, the quality inspection node and the acceptance node performed in sequence. The person to be operated can include the annotation personnel of the unauthorized project and the annotation personnel of the authorized project. Among them, the annotation personnel of the unauthorized project have weak annotation ability and participate in the annotation project for a short time. In this method, the corresponding learning parameters of each annotation personnel of the unauthorized project and each annotation personnel of the authorized project will be obtained first, that is, the learning ability values ​​of the annotation personnel of the unauthorized project and the authorized project.

[0053] In this embodiment, the above learning parameters are obtained by:

[0054] First, historical labeling tasks and historical labeling data of historical labeling tasks are collected. Then, the collected historical labeling tasks are compiled into a test set. Next, standard evaluation parameters are formed according to the historical node record data of the historical labeling tasks. Then, according to the standard evaluation parameters and the first processing situation recorded when each operator processes the historical labeling data of the historical labeling tasks of the first preset number of cycles, the processing ability of the operators is evaluated to obtain the learning parameters of each operator.

[0055] Among them, historical labeling tasks can be data labeling projects handled by operators who have been admitted to the project. These historical labeling tasks have corresponding historical labeling data, which can be understood as processed real operation data. Moreover, these historical labeling data have successfully passed each inspection node, and each inspection node also has corresponding recorded inspection data, that is, the historical node record data corresponding to the historical labeling task. At the same time, the historical node record data contains the standard labeling answers obtained by manual processing.

[0056] The historical annotation task also has the delivery information that it should have when it was processed, that is, the historical delivery information. Similarly, the historical delivery information includes all historical subtasks contained in the historical annotation task, the task types related to each other in the historical subtasks, the basic information of the historical annotation task of each historical subtask, the historical annotation task processing flow, the historical annotation task nodes, and the historical annotation task delivery cycle.

[0057] The historical record data obtained at each inspection node according to the historical annotation task can form a standardized evaluation value, that is, the above-mentioned standard evaluation parameter, which is used to evaluate the learning parameters of each operator in the future.

[0058] Considering the more accurate judgment of the learning parameters of the personnel to be operated and the training and improvement of their annotation ability before these personnel to be operated enter the project, in this embodiment, when obtaining the learning parameters, each personnel to be operated will cycle through the test set that is the historical annotation data for multiple times, so that the first processing situation recorded when the personnel to be operated process the historical annotation data of the historical annotation task for the first preset number of cycles can more accurately reflect their learning ability to annotate automatic data. It can be understood that each sub-parameter in the corresponding learning parameter is determined by taking the average value.

[0059] At the same time, this method adapts the annotation recognition algorithms of different scenarios, such as 2D box, 3D box, point cloud fusion and other annotation recognition algorithms. When the operator submits the practice results, the annotation accuracy during practice is automatically determined based on the algorithm. The final learning parameters are obtained after a more comprehensive evaluation of the operator's first processing situation based on the standard evaluation parameters.

[0060] Preferably, the method will adapt different judgment rules according to different annotation tools when judging the annotation accuracy of the operator during practice. For example: 2D box adapts to IOU algorithm; polygon / polyline similarity is suitable for Frechet distance measurement algorithm, that is, matching the shortest distance between each point in group A and all points in group B, and taking the maximum value; sun box / 3D box normalizes the distance of each group of matching points and takes the average value.

[0061] In this way, the test questions are evolved based on real historical operation data, and labelers who are not allowed to enter the project can be invited to repeatedly practice labeling of autonomous driving data. The AI ​​algorithm is integrated to automatically identify the test question practice results of labelers who are not allowed to enter the project. This learning parameter evaluation method does not require human participation in the evaluation, which is convenient for batch inspection of the operation quality of operators, saves the time cost of quality inspectors, and avoids the problem of misidentification during manual inspection.

[0062] The ultimate goal is to increase the familiarity of operators with the needs of autonomous driving data labeling. After meeting the familiarity requirements, the operators will automatically take over the formal data and start to perform the corresponding data labeling tasks.

[0063] It should be noted that, in this embodiment, when it is determined that the granularity of a subtask is smaller than a preset value, the subtasks with a granularity smaller than the preset value are merged according to the interrelated task types in the subtasks, the basic task information of each subtask, the task processing flow, the task nodes, and the task delivery cycle. Such a setting can prevent excessive minimum task units and can also perform logical mergers of task units, thereby improving labor efficiency and reducing individual learning costs.

[0064] S20, input the learning parameters into the trained task splitting model to obtain the labeling parameters output by the task splitting model. In this method, the labeling parameters are composed of two parts, one is the task processing capability evaluation parameter for the operator to handle the pending project that is about to be delivered to the customer, and the other is the operating capacity that each subtask or task unit of the pending project should have when being processed. More specifically, the task processing capability evaluation parameter includes the capability evaluation value of the operator, the timeliness evaluation value and the accuracy evaluation value. The timeliness evaluation value is obtained based on the timeliness value, and the accuracy evaluation value is obtained based on the accuracy value. The timeliness value forms a mapping relationship with the actual operation time, and the accuracy value forms a mapping relationship with the number of feedbacks of each check node of the historical labeling task.

[0065] The task splitting model provided in this method is trained based on the actual operation time and the number of feedbacks of each inspection node obtained by historical operators in handling historical annotation tasks, as well as the task splitting template formed based on the task flow scheduling engine.

[0066] Among them, the actual operation time obtained by historical operators in handling historical annotation tasks and the number of feedbacks at each inspection node are used to form the evaluation criteria for the task processing capability evaluation parameters required for subsequent task allocation, and the task splitting template is responsible for splitting tasks based on the customer's original needs and actual scenarios for the delivered projects, scheduling the split tasks according to priority, providing an annotation tool set, establishing inspection nodes, providing job flow procedures, and establishing forward and reverse operations between nodes.

[0067] The task flow scheduling engine includes a split setting module, a job initialization module, a job scheduling module, a labeling module, and a quality assurance module. The task splitting template formed based on the task flow scheduling engine also has basic capabilities such as automatic task scheduling, job flow engine, and data platform.

[0068] More specifically, the split setting module: since the information to be delivered contains the customer's original needs, this module will complete the standardized configuration of the demand elements based on the information to be delivered. The standardized configuration is to split different inspection objects and set them into universal templates. For example, the pull frame of a family car and the pull frame of a conical barrel are two independent operating units.

[0069] Job initialization module: Based on the standardized settings completed by the split setting module, the subtasks in the customer's to-be-delivered project are further split, several minimum executable labeling jobs are formed based on the original data, and the job capacity of each labeling job is determined.

[0070] Job scheduling module: Based on the preset rule engine, it automatically forms the scheduling priority between all scheduled jobs, such as rejected annotation jobs of each inspection node of high-priority data > new annotation jobs of high-priority data > rejected annotation jobs of each inspection node of other non-priority (the earlier the rejected tasks become rework tasks, the higher the priority) > other new annotation jobs without priority, and schedules the annotation jobs.

[0071] Annotation module: responsible for providing an annotation tool set, maintaining independent annotation pages, and supporting relevant operators to use annotation tools to complete split annotation tasks.

[0072] Quality Assurance Module: Responsible for the maintenance and creation of inspection processes for inspection nodes, and supports the extraction of qualified annotation jobs from the pool of all jobs to be inspected through business rules and their assignment to quality inspectors.

[0073] Flow establishment module: In this embodiment, the workflow solution can be set up based on Alibaba's open source process engine framework cola, and forward flow nodes can be established, such as marking → review → quality inspection → acceptance, etc., based on the built-in content type recognition rules for revisions, and the rejected target nodes can be dynamically determined during the reverse flow process of each node for reverse operation, that is, a reverse operation process between each node is established, and batch customized changes are supported, that is, the system automatically handles the crossed nodes.

[0074] It should be noted that according to the actual labeling needs of different customers and projects, different task splitting templates can be obtained by modifying the corresponding parameters in the task flow scheduling engine, forming a universal task splitting template that meets the project requirements. All autonomous driving labeling projects that require target recognition can be processed accordingly according to the task splitting template.

[0075] The operating capacity is determined by:

[0076] Obtain the processing status of historical annotation tasks at each checkpoint, and perform similarity matching between the to-be-delivered items and the historical annotation tasks based on the to-be-delivered information and the historical delivery information of the historical annotation tasks to obtain matching results. Then, predict the task difficulty of each subtask in the to-be-delivered items based on the processing status and matching results, and determine the job capacity of each annotation job based on the task difficulty of the subtasks and the splitting of the subtasks.

[0077] S30, based on the annotation parameters, perform task allocation for the project to be delivered. Since the job capacity of each annotation job obtained after splitting and the task processing capability evaluation parameters of each person to be worked on have been determined, the subsequent task allocation can be performed according to the job capacity of each annotation job and the job capacity of the subtask composed of at least one annotation job.

[0078] In this embodiment, when processing the autonomous driving data labeling related projects, multiple preset values ​​can be pre-set to form the first parameter weight, and each sub-parameter in the task processing capability evaluation parameter when the historical labeling task of the waiting operator is completed is compared with the corresponding preset values, so that the job priority of the waiting operator in the processing or scheduling of the pending project can be obtained accordingly. In some cases, such as when the comprehensive ability of the business personnel needs to be considered, the higher the job priority, the higher the priority of the waiting operator in the processing of the pending project.

[0079] Specifically, step S30 includes:

[0080] S31. Determine the task priority of each person to be operated according to the first parameter weight of each sub-parameter in the task processing capability evaluation parameter.

[0081] S32. Based on the job capacity of the labeling job, the subtask capacity of the subtask consisting of at least one labeling job, and the job priority of the person to be delivered, the items to be delivered are dispatched to the person to be delivered, with the condition that the labeling job and subtasks are completed on time and with quality. The subtask capacity is obtained by accumulating the job capacity of all labeling jobs to which the subtask belongs.

[0082] The timeliness evaluation value is mapped to the task processing capability of the corresponding personnel to be operated, reflecting the time management capability and work effort of the personnel to be operated when performing data annotation. The accuracy evaluation value is mapped to the professional capability of the corresponding personnel to be operated, reflecting the working experience and professional background knowledge of the personnel to be operated. In actual work, the progress of each split annotation job may deviate from expectations when being processed, resulting in the progress of the corresponding annotation job and the corresponding subtask failing to meet the delivery requirements. In this embodiment, the scheduling of the personnel to be operated will be carried out after the task is assigned, and ultimately the condition is to ensure that the annotation job and subtask are completed on time and with quality.

[0083] When making specific assignments, the task that is most suitable for the person to be delivered can be assigned to him according to his task processing ability evaluation parameters. This can not only reduce labor efficiency costs, shorten the time for the labeling personnel from the beginning of learning to finally getting started with the formal project from the original 1-2 weeks to less than 30 minutes, but also enable the operator to learn only one module in the operator learning stage, further reducing the learning cost of the labeling personnel. At the same time, this task allocation method gives priority to assigning labeling tasks to the personnel to be delivered who have high completion quality and fast completion speed, that is, giving priority to assigning tasks to personnel to be delivered with higher comprehensive capabilities, avoiding the backlog of subtasks and project quality that does not meet customer requirements, and ensuring that the projects to be delivered are delivered to customers on time and in quality.

[0084] Specifically, step S32 includes:

[0085] S321, evaluate the processing quality of the labeling job and the subtasks corresponding to the labeling job according to the information to be delivered, and determine the first probability of the unfinished labeling task being completed on time and with quality, and the second probability of the unfinished subtask being completed on time and with quality. Since the information to be delivered includes the basic task information, task processing flow and task delivery cycle of each subtask, and there is a mapping relationship between the completion quality and the processing time of the corresponding node, that is, the data labeling quality that should be displayed at the corresponding time, the processing quality of the labeling job and the subtasks corresponding to the labeling job can be evaluated through the information to be delivered.

[0086] The above-mentioned unfinished tasks refer to the labeling tasks that have not been started, have not been processed, and have not been checked by the check node, as well as the subtasks containing these labeling tasks.

[0087] S322. When it is determined that both the first probability and the second probability are not lower than the preset probability, it means that the labeling job and the subtasks corresponding to the labeling job can be completed on time and with quality based on the current task allocation method. Correspondingly, when it is determined that the first probability and / or the second probability is lower than the preset probability, the personnel to be operated are scheduled according to their job priority, the remaining job capacity of the unfinished labeling tasks and the remaining subtask capacity of the unfinished subtasks.

[0088] In this embodiment, scheduling includes replacing operators and adding operators. In some special cases, when there is excess production capacity, the current operators can also be reduced and arranged to perform new tasks.

[0089] The processing method of the perception data labeling task of the present invention obtains the learning parameters of each operator to be operated and the to-be-delivered information of the to-be-delivered project, and inputs these two data into the constructed task splitting model to obtain the labeling parameters output by the task splitting model, which is convenient for batch inspection of the operator quality, saves the time cost of the inspectors, and avoids the problem of misidentification during manual inspection. It provides a universal task splitting solution for data labeling of autonomous driving scenarios, greatly shortens the time for labeling personnel to start learning and finally get started with formal projects, and allocates tasks for the to-be-delivered projects through labeling parameters, then the labeling tasks are preferentially allocated to operators to be operated with high completion quality and fast completion speed, that is, the tasks are preferentially allocated to operators to be operated with higher comprehensive capabilities, ensuring that the to-be-delivered projects are delivered to customers on time and with quality.

[0090] In order to ensure the processing quality of the project, the processing capability of each operator should at least meet the minimum requirements of the project, that is, the learning parameters of each operator should not be lower than a certain mark. In this embodiment, the method further includes before step S20:

[0091] The learning parameters of each person to be operated are evaluated by means of a preset first evaluation standard and a preset second parameter weight of each sub-parameter in the preset parameters to determine whether the learning parameters are not lower than the preset parameters; if it is determined that the learning parameters are not lower than the preset parameters, step S20 is executed; if it is determined that the learning parameters are lower than the preset parameters, the historical annotation data of the historical annotation tasks for the second preset number of cycles are processed by the person to be operated; according to the annotation evaluation parameters and the second processing situation recorded when each person to be operated processes the historical annotation data of the historical annotation tasks for the second preset number of cycles, the processing capacity of the person to be operated is evaluated, the learning parameters of each person to be operated are updated, and it is determined again whether the updated learning parameters are not lower than the preset parameters; if they are still lower than the preset parameters, the test set for the second preset number of cycles is practiced again until the learning parameters of the corresponding person to be operated are not lower than the preset parameters.

[0092] In particular, after the learning parameters meet the minimum requirements of the project to process the project to be delivered, and after the project to be delivered is accepted by the customer, the project to be delivered can be used as a historical annotation task, and the project and various data generated during the processing can be added to the test question set to update the test question set.

[0093] The following describes an apparatus provided by an embodiment of the present invention. The apparatus described below and the method described above can refer to each other.

[0094] See also Figure 2 , Figure 2 A schematic diagram of the structure of a perception data labeling task device according to an embodiment of the present invention is shown, and the device may include:

[0095] The acquisition module 10 is used to obtain the learning parameters of each person to be operated and to obtain the information to be delivered of the project to be delivered. In this method, the information to be delivered includes all subtasks contained in the project to be delivered, the task types related to each other in the subtasks, the basic task information of each subtask, the task processing flow, the task node and the task delivery cycle. The learning parameters include the actual operation time of the person to be operated to process each historical annotation task and the number of feedbacks of each check node of the historical annotation task. The number of feedbacks is the rounds used when the current node passes. The actual operation time is the sum of the actual operation time corresponding to each round when the historical annotation task is completed. The check node includes the audit node, the quality inspection node and the acceptance node performed in sequence. The person to be operated can include the annotation personnel of the unauthorized project and the annotation personnel of the authorized project. Among them, the annotation personnel of the unauthorized project have weak annotation ability and participate in the annotation project for a short time. In this method, the corresponding learning parameters of each annotation personnel of the unauthorized project and each annotation personnel of the authorized project will be obtained first, that is, the learning ability values ​​of the annotation personnel of the unauthorized project and the authorized project.

[0096] In this embodiment, the above learning parameters are obtained by:

[0097] First, historical labeling tasks and historical labeling data of historical labeling tasks are collected. Then, the collected historical labeling tasks are compiled into a test set. Next, standard evaluation parameters are formed according to the historical node record data of the historical labeling tasks. Then, according to the standard evaluation parameters and the first processing situation recorded when each operator processes the historical labeling data of the historical labeling tasks of the first preset number of cycles, the processing ability of the operators is evaluated to obtain the learning parameters of each operator.

[0098] Among them, historical labeling tasks can be data labeling projects that have been handled by approved operators. These historical labeling tasks have corresponding historical labeling data, which can be understood as processed real operation data. Moreover, these historical labeling data have successfully passed each inspection node, and each inspection node also has corresponding recorded inspection data, that is, the historical node record data corresponding to the historical labeling task. At the same time, the historical node record data contains the standard labeling answers obtained by manual processing.

[0099] The historical annotation task also has the delivery information that it should have when it was processed, that is, the historical delivery information. Similarly, the historical delivery information includes all historical subtasks contained in the historical annotation task, the task types related to each other in the historical subtasks, the basic information of the historical annotation task of each historical subtask, the historical annotation task processing flow, the historical annotation task nodes, and the historical annotation task delivery cycle.

[0100] The historical record data obtained at each inspection node according to the historical annotation task can form a standardized evaluation value, that is, the above-mentioned standard evaluation parameter, which is used to evaluate the learning parameters of each operator in the future.

[0101] In order to more accurately judge the learning parameters of the personnel to be operated and to train and improve their labeling ability before they are admitted to the project, in this embodiment, when obtaining the learning parameters, each personnel to be operated will practice the test questions set multiple times in a cycle, that is, the historical labeling data. In this way, the first processing situation recorded when the personnel to be operated processes the historical labeling data of the historical labeling task for the first preset number of cycles can more accurately reflect their learning ability to label automatic data.

[0102] At the same time, this method adapts the annotation recognition algorithms of different scenarios, such as 2D box, 3D box, point cloud fusion and other annotation recognition algorithms. When the operator submits the practice results, the annotation accuracy during practice is automatically determined based on the algorithm. The final learning parameters are obtained after a more comprehensive evaluation of the operator's first processing situation based on the standard evaluation parameters.

[0103] In this way, the real historical operation data is evolved into test questions, and labelers who are not allowed to enter the project can be invited to repeatedly practice labeling of autonomous driving data. The AI ​​algorithm is integrated to automatically identify the test question practice results of labelers who are not allowed to enter the project. Ultimately, the familiarity of the operators with the autonomous driving data labeling requirements is improved. After the familiarity requirements are met, the operators automatically take over the formal data and start to perform the corresponding data labeling tasks.

[0104] It should be noted that, in this embodiment, when it is determined that the granularity of a subtask is smaller than a preset value, the subtasks with a granularity smaller than the preset value are merged according to the interrelated task types in the subtasks, the basic task information of each subtask, the task processing flow, the task nodes, and the task delivery cycle. Such a setting can prevent excessive minimum task units and can also perform logical mergers of task units, thereby improving labor efficiency and reducing individual learning costs.

[0105] The splitting module 20 is used to input the learning parameters into the trained task splitting model to obtain the labeling parameters output by the task splitting model. In this method, the labeling parameters are composed of two parts, one is the task processing capability evaluation parameter for the operator to handle the to-be-delivered project that is about to be delivered to the customer, and the other is the operating capacity that each subtask or task unit of the to-be-delivered project should have when being processed. More specifically, the task processing capability evaluation parameter includes the three parameters of the ability evaluation value of the operator, the timeliness evaluation value and the accuracy evaluation value. The timeliness evaluation value is obtained based on the timeliness value, and the accuracy evaluation value is obtained based on the accuracy value. The timeliness value forms a mapping relationship with the actual operation time, and the accuracy value forms a mapping relationship with the number of feedbacks of each check node of the historical labeling task.

[0106] The task splitting model provided in this method is trained based on the actual operation time and the number of feedbacks of each inspection node obtained by historical operators in handling historical annotation tasks, as well as the task splitting template formed based on the task flow scheduling engine.

[0107] Among them, the actual operation time obtained by historical operators in handling historical annotation tasks and the number of feedbacks at each inspection node are used to form the evaluation criteria for the task processing capability evaluation parameters required for subsequent task allocation, and the task splitting template is responsible for splitting tasks based on the customer's original needs and actual scenarios for the delivered projects, scheduling the split tasks according to priority, providing an annotation tool set, establishing inspection nodes, providing job flow procedures, and establishing forward and reverse operations between nodes.

[0108] The task flow scheduling engine includes a split setting module, a job initialization module, a job scheduling module, a labeling module, and a quality assurance module. The task splitting template formed based on the task flow scheduling engine also has basic capabilities such as automatic task scheduling, job flow engine, and data platform.

[0109] More specifically, the split setting module: since the information to be delivered contains the customer's original needs, this module will complete the standardized configuration of the demand elements based on the information to be delivered. The standardized configuration is to split different inspection objects and set them into universal templates. For example, the pull frame of a family car and the pull frame of a conical barrel are two independent operating units.

[0110] Job initialization module: Based on the standardized settings completed by the split setting module, the subtasks in the customer's to-be-delivered project are further split, several minimum executable labeling jobs are formed based on the original data, and the job capacity of each labeling job is determined.

[0111] Job scheduling module: Based on the preset rule engine, it automatically forms the scheduling priority between all scheduled jobs, such as rejected annotation jobs of each inspection node of high-priority data > new annotation jobs of high-priority data > rejected annotation jobs of each inspection node of other non-priority (the earlier the rejected tasks become rework tasks, the higher the priority) > other new annotation jobs without priority, and schedules the annotation jobs.

[0112] Annotation module: responsible for providing an annotation tool set, maintaining independent annotation pages, and supporting relevant operators to use annotation tools to complete split annotation tasks.

[0113] Quality Assurance Module: Responsible for the maintenance and creation of inspection processes for inspection nodes, and supports the extraction of qualified annotation jobs from the pool of all jobs to be inspected through business rules and their assignment to quality inspectors.

[0114] Flow establishment module: In this embodiment, the workflow solution can be set up based on Alibaba's open source process engine framework cola, and forward flow nodes can be established, such as marking → review → quality inspection → acceptance, etc., based on the built-in content type recognition rules for revisions, and the rejected target nodes can be dynamically determined during the reverse flow process of each node for reverse operation, that is, a reverse operation process between each node is established, and batch customized changes are supported, that is, the system automatically handles the crossed nodes.

[0115] It should be noted that according to the actual labeling needs of different customers and projects, different task splitting templates can be obtained by modifying the corresponding parameters in the task flow scheduling engine, forming a universal task splitting template that meets the project requirements. All autonomous driving labeling projects that require target recognition can be processed accordingly according to the task splitting template.

[0116] The allocation module 30 is used to allocate tasks for the projects to be delivered based on the annotation parameters. Since the operation capacity of each annotation job obtained after the splitting and the task processing ability evaluation parameters of each person to be operated have been determined, the subsequent task allocation can be carried out according to the operation capacity of each annotation job and the operation capacity of the subtask composed of at least one annotation job.

[0117] In this embodiment, when processing the autonomous driving data labeling related projects, multiple preset values ​​can be pre-set to form the first parameter weight, and each sub-parameter in the task processing capability evaluation parameter when the historical labeling task of the waiting operator is completed is compared with the corresponding preset values, so that the job priority of the waiting operator in the processing or scheduling of the pending project can be obtained accordingly. In some cases, such as when the comprehensive ability of the business personnel needs to be considered, the higher the job priority, the higher the priority of the waiting operator in the processing of the pending project.

[0118] The processing device of the perception data labeling task of the present invention obtains the learning parameters of each operator to be operated and the to-be-delivered information of the to-be-delivered project, and inputs these two data into the constructed task splitting model to obtain the labeling parameters output by the task splitting model, which is convenient for batch inspection of the operator quality, saves the time cost of the inspectors, and avoids the problem of misidentification during manual inspection. It provides a universal task splitting solution for data labeling of autonomous driving scenarios, greatly shortens the time for labeling personnel to start learning and finally get started with formal projects. By allocating tasks for the to-be-delivered projects through labeling parameters, labeling tasks are preferentially allocated to operators to be operated with high completion quality and fast completion speed, that is, tasks are preferentially allocated to operators to be operated with higher comprehensive capabilities, ensuring that the to-be-delivered projects are delivered to customers on time and with quality.

[0119] Figure 3 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 3 As shown, the electronic device may include: a processor 310 (processor), a communication interface 320 (Communications Interface), a memory 330 (memory) and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic command in the memory 330 to execute the processing method of the perception data labeling task, and the method includes:

[0120] Obtain the learning parameters of each person to be operated and the to-be-delivered information of the to-be-delivered project; the to-be-delivered information includes all subtasks included in the to-be-delivered project, the interrelated task types in the subtasks, the basic task information of each subtask, the task processing flow, the task node and the task delivery cycle, and, when it is determined that the granularity of the subtask is less than the preset value, merge the subtasks with a granularity less than the preset value according to the interrelated task types in the subtasks, the basic task information of each subtask, the task processing flow, the task node and the task delivery cycle; the learning parameters include the actual operation time of the person to be operated to process each historical annotation task and the number of feedbacks for each inspection node of the historical annotation task, the number of feedbacks is the rounds used when the current node passes, the actual operation time is the sum of the actual operation time corresponding to each round when the historical annotation task is completed, and the inspection nodes include the audit nodes, quality inspection nodes and acceptance nodes that are performed sequentially;

[0121] The learning parameters and the information to be delivered are input into the trained task splitting model to obtain the labeling parameters output by the task splitting model; the labeling parameters include the task processing capability evaluation parameters of the personnel to be operated to handle the items to be delivered, and the operating capacity required for the labeling job formed after each subtask of the items to be delivered is split; the task processing capability evaluation parameters include the capability evaluation value, timeliness evaluation value and accuracy evaluation value of the personnel to be operated; the timeliness evaluation value is obtained based on the timeliness value, the accuracy evaluation value is obtained based on the accuracy value, the timeliness value forms a mapping relationship with the actual operation time, and the accuracy value forms a mapping relationship with the number of feedbacks of each inspection node of the historical labeling task; the task splitting model is obtained based on the actual operation time and the number of feedbacks of each inspection node obtained by the historical operators in handling the historical labeling tasks, and the task splitting template formed based on the task flow scheduling engine is trained;

[0122] Based on the marking parameters, task allocation of the project to be delivered is performed.

[0123] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memor), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0124] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, and when the program instructions are executed by a computer, the computer can execute the processing method of the perception data labeling task provided by the above methods, the method comprising:

[0125] Obtain the learning parameters of each person to be operated and the to-be-delivered information of the to-be-delivered project; the to-be-delivered information includes all subtasks included in the to-be-delivered project, the interrelated task types in the subtasks, the basic task information of each subtask, the task processing flow, the task node and the task delivery cycle, and, when it is determined that the granularity of the subtask is less than the preset value, merge the subtasks with a granularity less than the preset value according to the interrelated task types in the subtasks, the basic task information of each subtask, the task processing flow, the task node and the task delivery cycle; the learning parameters include the actual operation time of the person to be operated to process each historical annotation task and the number of feedbacks for each inspection node of the historical annotation task, the number of feedbacks is the rounds used when the current node passes, the actual operation time is the sum of the actual operation time corresponding to each round when the historical annotation task is completed, and the inspection nodes include the audit nodes, quality inspection nodes and acceptance nodes that are performed sequentially;

[0126] The learning parameters and the information to be delivered are input into the trained task splitting model to obtain the labeling parameters output by the task splitting model; the labeling parameters include the task processing capability evaluation parameters of the personnel to be operated to handle the items to be delivered, and the operating capacity required for the labeling job formed after each subtask of the items to be delivered is split; the task processing capability evaluation parameters include the capability evaluation value, timeliness evaluation value and accuracy evaluation value of the personnel to be operated; the timeliness evaluation value is obtained based on the timeliness value, the accuracy evaluation value is obtained based on the accuracy value, the timeliness value forms a mapping relationship with the actual operation time, and the accuracy value forms a mapping relationship with the number of feedbacks of each inspection node of the historical labeling task; the task splitting model is obtained based on the actual operation time and the number of feedbacks of each inspection node obtained by the historical operators in handling the historical labeling tasks, and the task splitting template formed based on the task flow scheduling engine is trained;

[0127] Based on the marking parameters, task allocation of the project to be delivered is performed.

[0128] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the above-mentioned processing methods for performing the perception data labeling task, the method comprising:

[0129] Obtain the learning parameters of each person to be operated and the to-be-delivered information of the to-be-delivered project; the to-be-delivered information includes all subtasks included in the to-be-delivered project, the interrelated task types in the subtasks, the basic task information of each subtask, the task processing flow, the task node and the task delivery cycle, and, when it is determined that the granularity of the subtask is less than the preset value, merge the subtasks with a granularity less than the preset value according to the interrelated task types in the subtasks, the basic task information of each subtask, the task processing flow, the task node and the task delivery cycle; the learning parameters include the actual operation time of the person to be operated to process each historical annotation task and the number of feedbacks for each inspection node of the historical annotation task, the number of feedbacks is the rounds used when the current node passes, the actual operation time is the sum of the actual operation time corresponding to each round when the historical annotation task is completed, and the inspection nodes include the audit nodes, quality inspection nodes and acceptance nodes that are performed sequentially;

[0130] The learning parameters and the information to be delivered are input into the trained task splitting model to obtain the labeling parameters output by the task splitting model; the labeling parameters include the task processing capability evaluation parameters of the personnel to be operated to handle the items to be delivered, and the operating capacity required for the labeling job formed after each subtask of the items to be delivered is split; the task processing capability evaluation parameters include the capability evaluation value, timeliness evaluation value and accuracy evaluation value of the personnel to be operated; the timeliness evaluation value is obtained based on the timeliness value, the accuracy evaluation value is obtained based on the accuracy value, the timeliness value forms a mapping relationship with the actual operation time, and the accuracy value forms a mapping relationship with the number of feedbacks of each inspection node of the historical labeling task; the task splitting model is obtained based on the actual operation time and the number of feedbacks of each inspection node obtained by the historical operators in handling the historical labeling tasks, and the task splitting template formed based on the task flow scheduling engine is trained;

[0131] Based on the marking parameters, task allocation of the project to be delivered is performed.

[0132] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for processing a perceptual data labeling task, characterized in that: The method comprises: Obtain the learning parameters of each person to be operated and the to-be-delivered information of the to-be-delivered project; the to-be-delivered information includes all subtasks included in the to-be-delivered project, the interrelated task types in the subtasks, the basic task information of each subtask, the task processing flow, the task node and the task delivery cycle, and, when it is determined that the granularity of the subtask is less than the preset value, merge the subtasks with a granularity less than the preset value according to the interrelated task types in the subtasks, the basic task information of each subtask, the task processing flow, the task node and the task delivery cycle; the learning parameters include the actual operation time of the person to be operated to process each historical annotation task and the number of feedbacks for each inspection node of the historical annotation task, the number of feedbacks is the rounds used when the current node passes, the actual operation time is the sum of the actual operation time corresponding to each round when the historical annotation task is completed, and the inspection nodes include the audit nodes, quality inspection nodes and acceptance nodes that are performed sequentially; The learning parameters and the information to be delivered are input into the trained task splitting model to obtain the labeling parameters output by the task splitting model; the labeling parameters include the task processing capability evaluation parameters of the personnel to be operated to handle the items to be delivered, and the operating capacity required for the labeling job formed after each subtask of the items to be delivered is split; the task processing capability evaluation parameters include the capability evaluation value, timeliness evaluation value and accuracy evaluation value of the personnel to be operated; the timeliness evaluation value is obtained based on the timeliness value, the accuracy evaluation value is obtained based on the accuracy value, the timeliness value forms a mapping relationship with the actual operation time, and the accuracy value forms a mapping relationship with the number of feedbacks of each inspection node of the historical labeling task; the task splitting model is obtained based on the actual operation time and the number of feedbacks of each inspection node obtained by the historical operators in handling the historical labeling tasks, and the task splitting template formed based on the task flow scheduling engine is trained; Based on the annotation parameters, perform task allocation for the project to be delivered; The task allocation of the project to be delivered based on the marked parameters specifically includes: Determine the job priority of each person waiting to be operated according to the first parameter weight of each sub-parameter in the task processing capability evaluation parameter; Based on the job capacity of the labeling job, the subtask capacity of the subtask consisting of at least one labeling job, and the job priority of the person to be delivered, the items to be delivered are dispatched to the person to be delivered, with the condition that the labeling job and subtasks are completed on time and with quality; the subtask capacity is obtained by accumulating the job capacity of all labeling jobs to which the subtask belongs.

2. The method for processing the perceptual data annotation task according to claim 1, characterized in that: The learning parameters are obtained by following the steps below: Collect historical annotation tasks and historical annotation data of historical annotation tasks; Form standard evaluation parameters based on the historical node record data of the historical annotation task; According to the labeling evaluation parameters and the first processing situation recorded when each operator handles the historical labeling data of the historical labeling task of the first preset number of cycles, the processing ability of the operator is evaluated to obtain the learning parameters of each operator; the historical labeling data is the inspection data recorded when the historical labeling task passes through each inspection node.

3. The method for processing a perceptual data annotation task according to claim 1, characterized in that: The method of dispatching the items to be delivered to the personnel to be delivered based on the job capacity of the labeling job, the subtask capacity of the subtask composed of at least one labeling job, and the job priority of the personnel to be delivered, with the condition of ensuring that the labeling job and the subtask are completed on time and with quality, specifically includes: Evaluate the processing quality of the labeling job and the subtasks corresponding to the labeling job according to the information to be delivered, and determine a first probability that the unfinished labeling task will be completed on time and with quality, and a second probability that the unfinished subtask will be completed on time and with quality; When it is determined that the first probability and / or the second probability is lower than the preset probability, the personnel to be operated are scheduled according to their job priority, the remaining job capacity of the unfinished marking tasks, and the remaining subtask capacity of the unfinished subtasks.

4. The method for processing a perceptual data annotation task according to claim 1, characterized in that: The task flow scheduling engine includes a split setting module, a job initialization module, a job scheduling module, a marking module and a quality assurance module; The split setting module is used to complete the standardized configuration of the demand elements according to the information to be delivered to the customer, and the standardized configuration is to split different detection objects; The job initialization module is used to split the subtasks in the to-be-delivered project based on the standardized settings completed by the split setting module to form a number of labeling jobs; The job scheduling module is used to form the scheduling priority among all scheduled jobs based on the preset rule engine, and schedule the labeling jobs; The annotation module is used to provide an annotation tool set and maintain an independent annotation page; the tool set is used to provide the operator with the annotation work to complete the splitting of the annotation; The quality assurance module is used to maintain and create inspection processes for inspection nodes; Flow establishment module: used to set up workflow plans, establish forward flow nodes and reverse operation processes between nodes.

5. The method for processing the perceptual data labeling task according to claim 4, characterized in that: The operating capacity is determined by: Obtain the processing status of historical annotation tasks at each checkpoint; Perform similarity matching between the items to be delivered and the historical annotation tasks based on the information to be delivered and the historical delivery information of the historical annotation tasks to obtain a matching result; The difficulty of each subtask in the project to be delivered is predicted based on the processing status and matching results, and the job capacity of each labeling job is determined based on the difficulty of the subtask and the splitting of the subtask.

6. The method for processing a perceptual data annotation task according to claim 1, characterized in that: Before the step of inputting the learning parameters and the information to be delivered into the trained task splitting model to obtain the labeling parameters output by the task splitting model, the method further includes: The learning parameters of each worker to be operated are evaluated by a preset first evaluation standard and a preset second parameter weight of each sub-parameter in the preset parameter to determine whether the learning parameter is not lower than the preset parameter; When it is determined that the learning parameters are lower than the preset parameters, the historical labeling data of the historical labeling tasks for the second preset number of cycles are handed over to the personnel to be operated. According to the labeling evaluation parameters and the second processing situation recorded when each personnel to be operated processes the historical labeling data of the historical labeling tasks for the second preset number of cycles, the processing capabilities of the personnel to be operated are evaluated, and the learning parameters of each personnel to be operated are updated until the learning parameters of the corresponding personnel to be operated are no lower than the preset parameters.

7. A processing device for perceptual data labeling tasks, characterized in that: The device comprises: An acquisition module is used to acquire the learning parameters of each person to be operated and the to-be-delivered information of the to-be-delivered project; the to-be-delivered information includes all subtasks included in the to-be-delivered project, the interrelated task types in the subtasks, the basic task information of each subtask, the task processing flow, the task node and the task delivery cycle, and, when it is determined that the granularity of the subtask is less than the preset value, the subtasks with a granularity less than the preset value are merged according to the interrelated task types in the subtasks, the basic task information of each subtask, the task processing flow, the task node and the task delivery cycle; the learning parameters include the actual operation time consumed by the person to be operated to process each historical annotation task and the number of feedbacks for each inspection node of the historical annotation task, the number of feedbacks is the rounds used when the current node passes, the actual operation time is the sum of the actual operation time corresponding to each round when the historical annotation task is completed, and the inspection nodes include the audit nodes, quality inspection nodes and acceptance nodes that are performed sequentially; A splitting module is used to input the learning parameters and the information to be delivered into a trained task splitting model to obtain the labeling parameters output by the task splitting model; the labeling parameters include the task processing capability evaluation parameters of the personnel to be operated to handle the items to be delivered, and the operating capacity required for the labeling job formed after each subtask of the items to be delivered is split; the task processing capability evaluation parameters include the capability evaluation value, timeliness evaluation value and accuracy evaluation value of the personnel to be operated; the timeliness evaluation value is obtained based on the timeliness value, the accuracy evaluation value is obtained based on the accuracy value, the timeliness value forms a mapping relationship with the actual operation time, and the accuracy value forms a mapping relationship with the number of feedbacks of each inspection node of the historical labeling task; the task splitting model is trained based on the actual operation time and the number of feedbacks of each inspection node obtained by the historical operators in handling the historical labeling tasks, and the task splitting template formed based on the task flow scheduling engine; An allocation module, used for allocating tasks of the to-be-delivered project based on the annotation parameters; wherein the higher the value of the annotation parameter corresponding to the to-be-operated personnel is, the higher the priority of the to-be-operated personnel in processing the perception data annotation task is; The allocation module specifically includes: Determine the job priority of each person waiting to be operated according to the first parameter weight of each sub-parameter in the task processing capability evaluation parameter; Based on the job capacity of the labeling job, the subtask capacity of the subtask consisting of at least one labeling job, and the job priority of the person to be delivered, the items to be delivered are dispatched to the person to be delivered, with the condition that the labeling job and subtasks are completed on time and with quality; the subtask capacity is obtained by accumulating the job capacity of all labeling jobs to which the subtask belongs.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method for processing the perception data labeling task as described in any one of claims 1 to 6 are implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for processing the perception data labeling task as described in any one of claims 1 to 6 are implemented.

Images