A method, device, equipment and medium for labeling data of autonomous driving data

Abstract

The present invention discloses a data labeling method, device, equipment and medium for autonomous driving data, and relates to the field of autonomous driving. The method includes: obtaining the historical tasks of all data labeling processed by each candidate labeler, and obtaining the net recommendation degree of data labeling of each candidate labeler according to the processing status of each historical task; performing task decomposition, predicting the task difficulty of the task to be scheduled, and determining the target capacity of the project to be delivered; obtaining the predicted capacity of the candidate labeler according to the net recommendation degree of each candidate labeler and the task difficulty of the task to be scheduled; performing pre-scheduling of the project to be delivered, and allocating labeling personnel in the previous operation batch for the project to be delivered; performing dynamic scheduling of the project to be delivered, and adjusting the labeling personnel in the previous operation batch for the project to be delivered. The present invention greatly reduces the interference of human experience, and can be dynamically adjusted in combination with subsequent actual conditions, so as to ensure the on-time delivery of the project.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a data labeling method, device, equipment and medium for autonomous driving data. Background Art

[0002] With the rapid development of the field of autonomous driving, the requirements for training data of various neural network models required for autonomous driving scenarios are also increasing. Data labeling can help these models improve recognition accuracy and precision. At present, data labeling requires professional labelers in the field, especially in the field of autonomous driving, which has high standards for data labeling. Therefore, the labeled data needs to undergo quality inspection and processing at various inspection nodes such as review, quality inspection, and acceptance. If the results of the review, quality inspection, and acceptance do not meet the standards, the labelers need to process the data accordingly to correct the errors.

[0003] Usually, data demanders will require data processors to deliver a certain amount of labeled and high-quality data within a certain period of time, so it is necessary to control the project progress. At present, the main means of project progress management is to rely on the manual control of project managers, including personnel allocation, progress management, etc. However, due to the complexity of autonomous driving scenarios, diverse data formats, large differences in the capabilities of labelers, and different project cycles, it is difficult to allocate labelers and control project progress. Current project managers usually rely on manual experience to allocate labelers, which may lead to many problems and defects such as incorrect allocation of personnel, which undoubtedly poses a great challenge to the overall progress of data labeling projects.

[0004] Therefore, it is necessary to provide a method for data labeling for autonomous driving data in order to correctly allocate labeling personnel and produce high-quality data required for the autonomous driving model. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a data labeling method, device, equipment and medium for autonomous driving data, so as to solve the problem of incorrect allocation of personnel when allocating data labeling personnel based on experience in the field of autonomous driving.

[0006] According to a first aspect, an embodiment of the present invention provides a data labeling method for autonomous driving data, the method comprising:

[0007] Obtain all historical data annotation tasks handled by each candidate annotator, and obtain the net recommendation degree of each candidate annotator's data annotation based on the processing status of each historical task at each check node;

[0008] Decompose historical tasks and tasks to be scheduled, predict the difficulty of tasks to be scheduled based on the processing status of historical tasks at each inspection node, and determine the target capacity of the projects to be delivered based on the difficulty of tasks to be scheduled;

[0009] Obtaining a predicted capacity of each candidate labeler according to the net recommendation degree of each candidate labeler and the task difficulty of the task to be scheduled;

[0010] Determine the response time of each candidate annotator, pre-schedule the projects to be delivered based on the predicted capacity and response time of the candidate annotators, and assign annotators in the previous batch to the projects to be delivered;

[0011] Determine the actual completion status of the projects to be delivered, and if the completion status does not meet expectations, dynamically schedule the projects to be delivered based on the total predicted capacity of the labelers in the previous work batch and the predicted capacity of the candidate labelers, and adjust the labelers in the previous work batch for the projects to be delivered.

[0012] In combination with the first aspect, in the first implementation of the first aspect, obtaining all historical tasks of data labeling processed by each candidate labeler, and obtaining the net recommendation degree of data labeling of each candidate labeler according to the processing status of each historical task at each inspection node, specifically includes:

[0013] Obtain the time from now of all historical tasks of data annotation handled by each candidate annotator, and obtain the confidence weight of each historical task of each candidate annotator based on the time from now;

[0014] Decompose each historical task according to each check node, determine the quality accuracy of the node feedback and the total number of node feedback of the decomposed historical task at each check node, and determine the equivalent accuracy of each historical task at each check node according to the quality accuracy and the total number of node feedback;

[0015] Determine the constraint coefficient corresponding to each check node according to the equivalent accuracy rate, normalize the constraint coefficient, weight the normalized constraint coefficient and the equivalent accuracy rate to obtain the equivalent quality of each historical task, and obtain the confidence quality of each historical task according to the confidence weight and equivalent quality of each historical task;

[0016] Determine the capacity efficiency of each historical task, and obtain the annotation quality of each historical task according to the confidence quality capacity efficiency of each historical task;

[0017] The average of the annotation quality of all historical tasks handled by each candidate annotator is calculated to obtain the net recommendation degree of each candidate annotator.

[0018] In combination with the first implementation manner of the first aspect, in the second implementation manner of the first aspect, the obtaining of all historical tasks of data annotation processed by each candidate annotator, and obtaining the net recommendation degree of data annotation of each candidate annotator according to the processing status of each historical task at each inspection node, further includes:

[0019] The historical tasks are screened to remove the historical tasks that have not gone through a complete inspection node; the inspection node includes a review node, a quality inspection node, and an acceptance node that are performed sequentially.

[0020] In combination with the first aspect, in a third implementation of the first aspect, the optimization goal of the pre-scheduling is:

[0021]

[0022] in, represents the optimization target of the pre-scheduling; Indicates the total number of pre-scheduled labelers; Indicates the average response time of the pre-scheduled labelers;

[0023] The constraints of the pre-scheduling are:

[0024]

[0025]

[0026] in, Indicates The predicted capacity of the pre-scheduled labelers; Indicates the target capacity of the project to be delivered; Indicates The net recommendation of the pre-scheduled annotators; Indicates the preset value.

[0027] In combination with the third implementation manner of the first aspect, in the fourth implementation manner of the first aspect, the optimization goal of the dynamic scheduling is:

[0028]

[0029] in, represents the optimization target of the dynamic scheduling; Indicates the total number of dynamically scheduled annotation personnel; Indicates the average response time of dynamically scheduled labelers;

[0030] The constraints of the dynamic scheduling are:

[0031]

[0032]

[0033] in, Indicates The predicted capacity of the labeling personnel who are dynamically scheduled; Indicates The predicted capacity of the marking personnel in the current job batch; It represents the total number of labelers in the current job batch; and, when labelers need to be increased, the net recommendation degree of each dynamically scheduled labeler ≥ the preset value; when labelers need to be reduced, the net recommendation degree of ∄ dynamically scheduled labelers ≤ ∀ the net recommendation degree of unscheduled labelers.

[0034] In combination with the first aspect, in a fifth implementation of the first aspect, the method further includes, before the step of determining the response time of each candidate labeler, pre-scheduling the project to be delivered according to the predicted capacity and response time of the candidate labelers, and allocating labelers in the previous operation batch to the project to be delivered:

[0035] Determine the scheduling degree of each candidate labeler within a preset time, and exclude candidate labelers whose scheduling times exceed the preset degree according to the scheduling degree; the scheduling degree is the scheduling times / job task volume of the current batch scheduling ratio of the candidate labeler, and the current batch scheduling ratio is the scheduling times / batch task volume within the preset time.

[0036] In combination with the first aspect, in a sixth implementation of the first aspect, the method for determining the response time specifically includes:

[0037] The response time of the labeling personnel to be scheduled is determined according to the working status of the labeling personnel to be scheduled; the working status includes an idle state and a working state. The response time of the labeling personnel to be scheduled in the idle state is 0, and the response time of the labeling personnel to be scheduled in the working state is the remaining task output value / predicted capacity of the labeling personnel to be scheduled.

[0038] According to a second aspect, an embodiment of the present invention further provides a data labeling device for autonomous driving data, the device comprising:

[0039] The historical task processing module is used to obtain the historical tasks of all data annotations processed by each candidate annotator, and obtain the net recommendation degree of data annotation of each candidate annotator based on the processing status of each historical task at each check node;

[0040] The project capacity prediction module is used to decompose historical tasks and tasks to be scheduled, predict the difficulty of tasks to be scheduled based on the processing status of historical tasks at each inspection node, and determine the target capacity of the project to be delivered based on the difficulty of tasks to be scheduled;

[0041] A personnel capacity prediction module is used to obtain the predicted capacity of each candidate labeling personnel according to the net recommendation degree of each candidate labeling personnel and the task difficulty of the task to be scheduled;

[0042] The project pre-scheduling module is used to determine the response time of each candidate labeler, pre-scheduling the projects to be delivered based on the predicted capacity and response time of the candidate labelers, and assigning labelers in the previous batch to the projects to be delivered;

[0043] The project dynamic scheduling module is used to determine the actual completion status of the projects to be delivered. If the completion status does not meet expectations, the project to be delivered will be dynamically scheduled based on the total predicted capacity of the labeling personnel in the previous work batch and the predicted capacity of the candidate labeling personnel, and the labeling personnel in the previous work batch will be adjusted for the projects to be delivered.

[0044] According to the third aspect, an embodiment of the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the data labeling method for autonomous driving data as described in any one of the above-mentioned methods are implemented.

[0045] According to a fourth aspect, an embodiment of the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a data labeling method for autonomous driving data as described in any one of the above-mentioned methods.

[0046] The data labeling method, device, equipment and medium for autonomous driving data of the present invention obtain all historical tasks of data labeling handled by each candidate labeler, and obtain the net recommendation degree of data labeling of each candidate labeler based on the processing status of each historical task at each inspection node. Afterwards, the task difficulty of the task to be scheduled is predicted based on the processing status of the historical tasks at each inspection node, and the target capacity of the project to be delivered is determined based on the task difficulty of the task to be scheduled. Then, the predicted capacity of each candidate labeler is obtained based on the net recommendation degree of each candidate labeler and the task difficulty of the task to be scheduled. Finally, the scheduling strategy for labeling personnel scheduling combines the "pre-reaction scheduling mode" and the "dynamic scheduling mode" for personnel allocation. In this way, the comprehensive efficiency can be improved in the case of complex audit quality inspection and acceptance with uncertain rounds of rejection and return for repair. , quality and other factors to evaluate the labeling ability, that is, the operating ability of each candidate labeler, providing confidence information for the allocation of candidate labelers, and also greatly reducing the energy consumed by project managers in screening candidate labelers, improving the accuracy of automatic allocation, and then combining the advantages of "pre-reaction scheduling mode" and "dynamic scheduling mode" when allocating personnel for tasks to be delivered, which can fully identify the risks of project progress, redundant personnel shortages, etc., and can accurately dispatch the appropriate labelers for processing tasks combined with the net recommendation degree of the candidate labelers, so as to realize the allocation of candidate labelers under limited information, greatly reducing the interference of human experience, and can be dynamically adjusted according to the subsequent actual situation, without the need for human attention to the project progress, thereby ensuring the timely delivery of the project and further reducing the burden on operators. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The features and advantages of the present invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present invention in any way. In the accompanying drawings:

[0048] Figure 1 A schematic diagram showing a flow chart of a method for labeling data of autonomous driving data provided by the present invention;

[0049] Figure 2 A structural schematic diagram of a method for labeling data for autonomous driving data provided by the present invention is shown;

[0050] Figure 3 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0052] Autonomous driving technology is rapidly becoming the focus of the automotive industry. With the mature application of end-to-end technologies, the field of autonomous driving has become the main driving force for artificial intelligence innovation, driving artificial intelligence systems to handle complex and unpredictable real-world scenarios. The development of the autonomous driving industry has not only promoted technological innovation, but also brought users a safer and more convenient driving experience.

[0053] Technological innovation is the core driving force for the development of the autonomous driving industry. Breakthroughs in sensor technology, artificial intelligence algorithms, big data processing, and 5G communications will significantly improve the perception, decision-making, and control capabilities of autonomous driving systems. As these technologies continue to mature, autonomous driving vehicles will achieve a leap from assisted driving to fully unmanned driving, bringing revolutionary changes to transportation. With the rapid development of the field of autonomous driving, the requirements for training data for various neural network models required for autonomous driving scenarios are also increasing. Data labeling can help these models improve recognition accuracy and precision. At present, data labeling requires professional labelers in the field, especially in the field of autonomous driving, which has high standards for data labeling. Therefore, the labeled data also needs to undergo quality inspection and processing at various inspection nodes such as review, quality inspection, and acceptance. If the results of the review, quality inspection, and acceptance do not meet the standards, the labelers need to process the data accordingly to correct the errors.

[0054] Usually, data demanders will require data processors to deliver a certain amount of labeled and high-quality data within a certain period of time, so it is necessary to control the project progress. At present, the main means of project progress management is to rely on the manual control of project managers, including personnel allocation, progress management, etc. However, due to the complexity of autonomous driving scenarios, diverse data formats, large differences in the capabilities of labelers, and different project cycles, it is difficult to allocate labelers and control project progress. Current project managers usually rely on manual experience to allocate labelers, which may lead to many problems and defects such as incorrect allocation of personnel, which undoubtedly poses a great challenge to the overall progress of data labeling projects.

[0055] For example, when a project manager allocates personnel based on manual experience, the following may occur: (1) the capabilities of the labelers do not match the project requirements; (2) too many or too few labelers are allocated, and the required project does not have a reasonable number of labelers; (3) the labelers have other labeling work in progress, etc. These problems undoubtedly bring significant risks to the project progress. Although the project manager will also monitor the project progress, when the progress is too slow, he will still use his experience to add more labelers, and when the project progress is too fast, he will remove some labelers, which will cause some problems: (4) This low degree of automation consumes a lot of energy of the project manager, and (5) the project manager may not be able to monitor and handle it in time.

[0056] Therefore, it is necessary to provide a method for data labeling for autonomous driving data in order to correctly allocate labeling personnel and produce high-quality data required for the autonomous driving model.

[0057] In order to solve the above problems, a data labeling method for autonomous driving data is provided in this embodiment, aiming to better allocate the labeling personnel required for autonomous driving data labeling. The data labeling method for autonomous driving data in the embodiment of the present invention can be used in electronic devices, including but not limited to computers, mobile terminals, etc. Figure 1 is a flow chart of a method for labeling data of autonomous driving data according to an embodiment of the present invention. Figure 1 As shown, the method may include the following steps:

[0058] S10. Obtain all historical tasks of data labeling handled by each candidate labeler, and obtain the net recommendation degree of data labeling of each candidate labeler based on the processing status of each historical task at each check node.

[0059] Through the processing of step S10, the labeling ability of the candidate labelers to be scheduled can be objectively quantified, which will serve as a pre-processing for the subsequent candidate labeler allocation system. In step S10, the time since the historical tasks handled by the candidate labelers and the actual pass rate of the historical tasks at each check node will be comprehensively considered, thereby obtaining a more objective and accurate net recommendation degree.

[0060] Among them, the inspection nodes include audit nodes, quality inspection nodes and acceptance nodes which are carried out in sequence.

[0061] S20, decomposing historical tasks and tasks to be scheduled, predicting the difficulty of tasks to be scheduled based on the processing status of historical tasks at each inspection node, and determining the target capacity of the projects to be delivered based on the difficulty of tasks to be scheduled.

[0062] In this embodiment, based on the processing status of historical tasks at each inspection node, the historical tasks can be decomposed and the difficulty of the tasks to be scheduled can be predicted by combining the similarity between the new tasks to be scheduled and the historical tasks at each task node process.

[0063] Correspondingly, the target capacity of a delivery project consisting of multiple tasks to be scheduled is: the total task difficulty of all tasks to be scheduled / delivery cycle. Secondly, the total task difficulty of all tasks to be scheduled is obtained by accumulating the task difficulties of all tasks to be scheduled.

[0064] In this embodiment, the delivery cycle can be that after the project is started, batches are manually created and data is uploaded, and the expected delivery time is filled in.

[0065] S30: Obtain the predicted capacity of each candidate labeler according to the net recommendation degree of each candidate labeler and the difficulty of the task to be scheduled.

[0066] By traversing the net recommendation scores of the candidate labelers in the pool to be scheduled, the predicted capacity of each candidate labeler can be obtained.

[0067] In this embodiment, the predicted capacity of each labeling personnel is the net recommendation degree / task difficulty of the task to be scheduled.

[0068] S40: Determine the response time of each candidate labeler, pre-schedule the project to be delivered based on the predicted capacity and response time of the candidate labelers, and allocate labelers in the previous work batch to the project to be delivered.

[0069] In this embodiment, the expected response time of each candidate annotation personnel is predicted by his / her working status. Obviously, the response time of the idle annotation personnel to be scheduled is 0, while the response time of the annotation personnel to be scheduled in the working status is the remaining task output value / predicted capacity.

[0070] S50. Determine the actual completion status of the project to be delivered. If the completion status does not meet expectations, dynamically schedule the project to be delivered based on the total predicted capacity of the labeling personnel in the previous work batch and the predicted capacity of the candidate labeling personnel, and adjust the labeling personnel in the previous work batch for the project to be delivered.

[0071] In this embodiment, a scheduling method for labeling personnel is provided. This scheduling method can fully identify the risks of project progress, redundant personnel shortages, etc., and can accurately schedule labeling personnel suitable for handling tasks in combination with the net recommendation degree of candidate labeling personnel. In addition, the scheduling strategy during scheduling combines the "pre-reaction scheduling mode" and the "dynamic scheduling mode" for personnel allocation, thereby ensuring the on-time delivery of the project.

[0072] The data labeling method for autonomous driving data of the present invention obtains the historical tasks of all data labeling handled by each candidate labeler, and obtains the net recommendation degree of data labeling of each candidate labeler according to the processing status of each historical task at each inspection node. Afterwards, the task difficulty of the task to be scheduled is predicted according to the processing status of the historical tasks at each inspection node, and the target capacity of the project to be delivered is determined according to the task difficulty of the task to be scheduled. Then, the predicted capacity of each candidate labeler is obtained according to the net recommendation degree of each candidate labeler and the task difficulty of the task to be scheduled. Finally, the scheduling strategy for labeling personnel scheduling combines the "pre-reaction scheduling mode" and the "dynamic scheduling mode" for personnel allocation. In this way, under the condition of complex audit quality inspection and acceptance with uncertain rounds of rejection and return for repair, comprehensive efficiency, quality, etc. can be achieved. The factors evaluate the labeling ability, that is, the operating ability, of each candidate labeler, and provide confidence information for the allocation of candidate labelers. At the same time, it also greatly reduces the energy consumed by project managers in screening candidate labelers, and improves the accuracy of automatic allocation. Later, when allocating personnel for tasks to be delivered, it combines the advantages of "pre-reaction scheduling mode" and "dynamic scheduling mode", which can fully identify the risks of project progress, redundant personnel shortages, etc., and can accurately dispatch the appropriate labelers for processing tasks in combination with the net recommendation degree of the candidate labelers, so as to realize the allocation of candidate labelers under limited information, greatly reduce the interference of human experience, and can make dynamic adjustments based on subsequent actual conditions, without the need for human attention to the project progress, thereby ensuring the timely delivery of the project and further reducing the burden on operators.

[0073] Wherein, step S10 specifically includes:

[0074] S11. Obtain the time from now of all historical tasks of data labeling handled by each candidate labeler, and obtain the confidence weight of each historical task of each candidate labeler based on the time from now.

[0075] In this embodiment, we first search through the historical tasks of the candidate annotators (referred to as Task 1, Task 2, Task N, etc.). Since each historical task is different in time, the confidence level of its participation in quantification is also different. , and its confidence weight is ,in It is the preset time reference parameter.

[0076] Preferably, the time from the historical task to the present is the time difference between the start time of the historical task and the current time.

[0077] It should be noted that, in this embodiment, before determining the time from the present of the historical tasks, it is necessary to screen the historical tasks handled by the candidate annotation personnel to eliminate the historical tasks that cannot provide reference value. The specific screening criteria are that the historical tasks need to be passed at various quality inspection nodes such as review, quality inspection, and acceptance.

[0078] S12. Decompose each historical task according to each inspection node, determine the quality accuracy of the node feedback of the decomposed historical task at each inspection node and the total number of node feedbacks, and determine the equivalent accuracy of each historical task at each inspection node based on the quality accuracy and the total number of node feedbacks.

[0079] After that, each historical task is disassembled according to the check nodes, and its subsequent check nodes are disassembled. Specifically, the check nodes such as review, quality inspection, and acceptance will be checked according to objective standards. Each check node may have any number of rejections and passes. The quality accuracy of node feedback (including rejection and approval) is , the total number of node feedbacks for a single check node is Obviously, the accuracy rate corresponding to the feedback of the earlier nodes can better reflect the real annotation ability of the annotators. Therefore, the accuracy rate weight of the later times should be penalized, and finally the audit equivalent accuracy rate of each inspection node is obtained. , quality inspection equivalent accuracy And acceptance equivalent accuracy The calculation formula is:

[0080]

[0081] in, Indicates that the historical task is The equivalent accuracy of the check nodes.

[0082] S13. Determine the constraint coefficient corresponding to each check node according to the equivalent accuracy rate, normalize the constraint coefficient, weight the normalized constraint coefficient and the equivalent accuracy rate to obtain the equivalent quality of each historical task, and obtain the confidence quality of each historical task according to the confidence weight and equivalent quality of each historical task.

[0083] According to the actual pass rate of each inspection node, the constraint coefficients corresponding to the three inspection nodes of audit, quality inspection and acceptance are obtained. , , And normalized, where is the first constraint coefficient, is the constraint coefficient corresponding to the audit check node; is the second constraint coefficient, which is the constraint coefficient corresponding to the quality inspection node; is the third constraint coefficient, and is the constraint coefficient corresponding to the acceptance check node.

[0084] S14. Determine the capacity efficiency of each historical task, and obtain the labeling quality of each historical task according to the confidence quality capacity efficiency of each historical task.

[0085] S15. Calculate the average of the annotation quality of all historical tasks handled by each candidate annotator to obtain the net recommendation degree of each candidate annotator.

[0086] Then the normalized constraint coefficient Equivalent accuracy of each check node Weighted summation to obtain equivalent mass , and combined with the time confidence weight, we get the confidence quality of a single task, and multiply the confidence quality by the capacity efficiency of the task to get the labeling quality of the task. The average of the labeling quality of all tasks (task 1, task 2, task N, etc.) is the net recommendation of the labeler.

[0087] Wherein, step S40 specifically includes:

[0088] The optimization goal of pre-scheduling is:

[0089]

[0090] in, represents the optimization target of the pre-scheduling; Indicates the total number of pre-scheduled labelers; Indicates the average response time of the pre-scheduled labelers;

[0091] The constraints for pre-scheduling are:

[0092]

[0093]

[0094] in, Indicates The predicted capacity of the pre-scheduled labelers; Indicates the target capacity of the project to be delivered; Indicates The net recommendation of the pre-scheduled annotators; Indicates the preset value.

[0095] In this embodiment, the "pre-reaction scheduling mode" is to perform a pre-scheduling, and the optimization target of the pre-scheduling is min (the number of pre-scheduled labeling personnel × the average response time per person), and the constraints of the pre-scheduling are: the total predicted capacity of the pre-scheduled labeling personnel ≥ the target capacity of the projects to be delivered, and the net recommendation degree of each pre-scheduled labeling personnel ≥ the preset value.

[0096] Wherein, step S50 specifically includes:

[0097] In combination with the third implementation manner of the first aspect, in the fourth implementation manner of the first aspect, the optimization goal of the dynamic scheduling is:

[0098]

[0099] in, represents the optimization target of the dynamic scheduling; Indicates the total number of dynamically scheduled annotation personnel; Indicates the average response time of dynamically scheduled labelers;

[0100] The constraints of the dynamic scheduling are:

[0101]

[0102]

[0103] in, Indicates The predicted capacity of the labeling personnel who are dynamically scheduled; Indicates The predicted capacity of the marking personnel in the current job batch; Indicates the total number of labelers in the current job batch.

[0104] After the pre-scheduling is completed, the project to be delivered will start the actual operation. During the actual operation, the project progress may deviate from expectations. In this embodiment, the "dynamic scheduling mode" is implemented again. When the project progress is slower or much faster than expected, a dynamic scheduling is started, and this dynamic scheduling will recalibrate the production capacity within the batch. It is understandable that more than one dynamic scheduling may be required during the actual operation of the project to be delivered.

[0105] In this embodiment, the optimization goal of each dynamic scheduling is min (the number of dynamically scheduled labeling personnel × average response time per person), and the dynamic scheduling constraints are: ± the total predicted capacity of the dynamically scheduled labeling personnel + the total predicted capacity of the labeling personnel in the current job batch ≥ the target capacity of the project to be delivered, ± (the total predicted capacity of the pre-scheduled labeling personnel - the predicted capacity of any one dynamically scheduled labeling personnel) + the total predicted capacity of the labeling personnel in the current job batch < the target capacity of the project to be delivered, and the net recommendation degree of each dynamically scheduled labeling personnel ≥ the preset value (when the above ± is +, that is, when the labeling personnel need to be increased), the net recommendation degree of the dynamically scheduled labeling personnel ∄ ≤ the net recommendation degree of the unscheduled labeling personnel ∀ (when the above ± is -, that is, when the labeling personnel need to be reduced). By solving the optimization problem using dynamic programming, a specific list of scheduling personnel can be obtained.

[0106] In this way, after initiating a pre-scheduling, dynamic scheduling will be adopted when the subsequent project progress is abnormal. Obviously, high-frequency scheduling will lead to excessively high personnel turnover costs and increased overall batch learning costs. Considering the above problems, in this embodiment, a scheduling frequency penalty mechanism is also provided.

[0107] Accordingly, the method may further include the following steps before scheduling:

[0108] The scheduling degree of each candidate labeling personnel within a preset time is determined, and the candidate labeling personnel whose scheduling times exceed the preset degree are excluded according to the scheduling degree.

[0109] Preferably, the preset time can be configured as nearly n days.

[0110] Specifically, firstly, the number of scheduling times / job task volume of the current batch scheduling ratio of each labeling personnel in the past n days is calculated based on the traversal, where the current batch scheduling ratio = the number of scheduling times in the past n days / batch task volume, and the optimization goal is transformed into min (current batch scheduling ratio × new scheduling times + current personnel scheduling ratio × new scheduling times of personnel). In particular, the new scheduling times of all labeling personnel are obviously 1.

[0111] The following describes an apparatus provided by an embodiment of the present invention. The apparatus described below and the method described above can refer to each other.

[0112] See also Figure 2 , Figure 2 A schematic diagram showing the structure of a data labeling device for autonomous driving data according to an embodiment of the present invention is shown. The device may include:

[0113] The historical task processing module 10 is used to obtain all historical tasks of data labeling processed by each candidate labeler, and obtain the net recommendation degree of data labeling of each candidate labeler according to the processing status of each historical task at each inspection node.

[0114] By doing so, the labeling ability of the candidate labelers to be scheduled can be objectively quantified, which will serve as a pre-processing for the subsequent candidate labeler allocation system. In step S10, the time since the historical tasks handled by the candidate labelers and the actual pass rate of the historical tasks at each check point will be comprehensively considered to obtain a more objective and accurate net recommendation.

[0115] Among them, the inspection nodes include audit nodes, quality inspection nodes and acceptance nodes which are carried out in sequence.

[0116] The project capacity prediction module 20 is used to decompose historical tasks and tasks to be scheduled, predict the task difficulty of the tasks to be scheduled according to the processing status of the historical tasks at each inspection node, and determine the target capacity of the project to be delivered according to the task difficulty of the tasks to be scheduled.

[0117] In this embodiment, based on the processing status of historical tasks at each inspection node, the historical tasks can be decomposed and the difficulty of the tasks to be scheduled can be predicted by combining the similarity between the new tasks to be scheduled and the historical tasks at each task node process.

[0118] Correspondingly, the target capacity of a delivery project consisting of multiple tasks to be scheduled is: the total task difficulty of all tasks to be scheduled / delivery cycle. Secondly, the total task difficulty of all tasks to be scheduled is obtained by accumulating the task difficulties of all tasks to be scheduled.

[0119] In this embodiment, the delivery cycle can be that after the project is started, batches are manually created and data is uploaded, and the expected delivery time is filled in.

[0120] The personnel capacity prediction module 30 is used to obtain the predicted capacity of each candidate labeling personnel according to the net recommendation degree of each candidate labeling personnel and the task difficulty of the task to be scheduled.

[0121] By traversing the net recommendation scores of the candidate labelers in the pool to be scheduled, the predicted capacity of each candidate labeler can be obtained.

[0122] In this embodiment, the predicted capacity of each labeling personnel is the net recommendation degree / task difficulty of the task to be scheduled.

[0123] The project pre-scheduling module 40 is used to determine the response time of each candidate labeler, pre-scheduling the pending projects according to the predicted capacity and response time of the candidate labelers, and assigning labelers in the previous work batch to the pending projects.

[0124] In this embodiment, the expected response time of each candidate annotation personnel is predicted by his / her working status. Obviously, the response time of the idle annotation personnel to be scheduled is 0, while the response time of the annotation personnel to be scheduled in the working status is the remaining task output value / predicted capacity.

[0125] The project dynamic scheduling module 50 is used to determine the actual completion status of the project to be delivered, and if the completion status does not meet expectations, dynamically schedule the project to be delivered based on the total predicted capacity of the labeling personnel in the previous work batch and the predicted capacity of the candidate labeling personnel, and adjust the labeling personnel in the previous work batch for the project to be delivered.

[0126] In this embodiment, a scheduling method for labeling personnel is provided. This scheduling method can fully identify the risks of project progress, redundant personnel shortages, etc., and can accurately schedule labeling personnel suitable for handling tasks in combination with the net recommendation degree of candidate labeling personnel. In addition, the scheduling strategy during scheduling combines the "pre-reaction scheduling mode" and the "dynamic scheduling mode" for personnel allocation, thereby ensuring the on-time delivery of the project.

[0127] The data labeling device for autonomous driving data of the present invention obtains the historical tasks of all data labeling handled by each candidate labeler, and obtains the net recommendation degree of data labeling of each candidate labeler according to the processing status of each historical task at each inspection node. Afterwards, the task difficulty of the task to be scheduled is predicted according to the processing status of the historical tasks at each inspection node, and the target capacity of the project to be delivered is determined according to the task difficulty of the task to be scheduled. Then, the predicted capacity of each candidate labeler is obtained according to the net recommendation degree of each candidate labeler and the task difficulty of the task to be scheduled. Finally, the scheduling strategy for labeling personnel scheduling combines the "pre-reaction scheduling mode" and the "dynamic scheduling mode" for personnel allocation. In this way, under the condition of complex audit quality inspection and acceptance with uncertain rounds of rejection and return for repair, comprehensive efficiency, quality, etc. can be achieved. The factors evaluate the labeling ability, that is, the operating ability, of each candidate labeler, and provide confidence information for the allocation of candidate labelers. At the same time, it also greatly reduces the energy consumed by project managers in screening candidate labelers, and improves the accuracy of automatic allocation. Later, when allocating personnel for tasks to be delivered, it combines the advantages of "pre-reaction scheduling mode" and "dynamic scheduling mode", which can fully identify the risks of project progress, redundant personnel shortages, etc., and can accurately dispatch the appropriate labelers for processing tasks in combination with the net recommendation degree of the candidate labelers, so as to realize the allocation of candidate labelers under limited information, greatly reduce the interference of human experience, and can make dynamic adjustments based on subsequent actual conditions, without the need for human attention to the project progress, thereby ensuring the timely delivery of the project and further reducing the burden on operators.

[0128] Figure 3An example of a physical structure diagram of an electronic device is shown in FIG. Figure 3 As shown, the electronic device may include: a processor 310 (processor), a communication interface 320 (Communications Interface), a memory 330 (memory) and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic command in the memory 330 to execute the data labeling method of the autonomous driving data, and the method includes:

[0129] Obtain all historical data annotation tasks handled by each candidate annotator, and obtain the net recommendation degree of each candidate annotator's data annotation based on the processing status of each historical task at each check node;

[0130] Decompose historical tasks and tasks to be scheduled, predict the difficulty of tasks to be scheduled based on the processing status of historical tasks at each inspection node, and determine the target capacity of the projects to be delivered based on the difficulty of tasks to be scheduled;

[0131] Obtaining a predicted capacity of each candidate labeler according to the net recommendation degree of each candidate labeler and the task difficulty of the task to be scheduled;

[0132] Determine the response time of each candidate annotator, pre-schedule the projects to be delivered based on the predicted capacity and response time of the candidate annotators, and assign annotators in the previous batch to the projects to be delivered;

[0133] Determine the actual completion status of the projects to be delivered, and if the completion status does not meet expectations, dynamically schedule the projects to be delivered based on the total predicted capacity of the labelers in the previous work batch and the predicted capacity of the candidate labelers, and adjust the labelers in the previous work batch for the projects to be delivered.

[0134] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memor), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0135] On the other hand, the present invention further provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, and when the program instructions are executed by a computer, the computer can execute the data labeling method for autonomous driving data provided by the above methods, the method comprising:

[0136] Obtain all historical data annotation tasks handled by each candidate annotator, and obtain the net recommendation degree of each candidate annotator's data annotation based on the processing status of each historical task at each check node;

[0137] Decompose historical tasks and tasks to be scheduled, predict the difficulty of tasks to be scheduled based on the processing status of historical tasks at each inspection node, and determine the target capacity of the projects to be delivered based on the difficulty of tasks to be scheduled;

[0138] Obtaining a predicted capacity of each candidate labeler according to the net recommendation degree of each candidate labeler and the task difficulty of the task to be scheduled;

[0139] Determine the response time of each candidate annotator, pre-schedule the projects to be delivered based on the predicted capacity and response time of the candidate annotators, and assign annotators in the previous batch to the projects to be delivered;

[0140] Determine the actual completion status of the projects to be delivered, and if the completion status does not meet expectations, dynamically schedule the projects to be delivered based on the total predicted capacity of the labelers in the previous work batch and the predicted capacity of the candidate labelers, and adjust the labelers in the previous work batch for the projects to be delivered.

[0141] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for labeling data for autonomous driving data provided above is implemented, and the method comprises:

[0142] Obtain all historical data annotation tasks handled by each candidate annotator, and obtain the net recommendation degree of each candidate annotator's data annotation based on the processing status of each historical task at each check node;

[0143] Decompose historical tasks and tasks to be scheduled, predict the difficulty of tasks to be scheduled based on the processing status of historical tasks at each inspection node, and determine the target capacity of the projects to be delivered based on the difficulty of tasks to be scheduled;

[0144] Obtaining a predicted capacity of each candidate labeler according to the net recommendation degree of each candidate labeler and the task difficulty of the task to be scheduled;

[0145] Determine the response time of each candidate annotator, pre-schedule the projects to be delivered based on the predicted capacity and response time of the candidate annotators, and assign annotators in the previous batch to the projects to be delivered;

[0146] Determine the actual completion status of the projects to be delivered, and if the completion status does not meet expectations, dynamically schedule the projects to be delivered based on the total predicted capacity of the labelers in the previous work batch and the predicted capacity of the candidate labelers, and adjust the labelers in the previous work batch for the projects to be delivered.

[0147] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data labeling method for autonomous driving data, characterized in that: The method comprises: Obtain all historical data annotation tasks handled by each candidate annotator, and obtain the net recommendation degree of each candidate annotator's data annotation based on the processing status of each historical task at each check node; Decompose historical tasks and tasks to be scheduled, predict the difficulty of tasks to be scheduled based on the processing status of historical tasks at each inspection node, and determine the target capacity of the projects to be delivered based on the difficulty of tasks to be scheduled; Obtaining a predicted capacity of each candidate labeler according to the net recommendation degree of each candidate labeler and the task difficulty of the task to be scheduled; Determine the response time of each candidate annotator, pre-schedule the projects to be delivered based on the predicted capacity and response time of the candidate annotators, and assign annotators in the previous batch to the projects to be delivered; Determine the actual completion status of the projects to be delivered, and if the completion status does not meet expectations, dynamically schedule the projects to be delivered based on the total predicted capacity of the labelers in the previous batch and the predicted capacity of the candidate labelers, and adjust the labelers in the previous batch for the projects to be delivered; The method of obtaining all historical tasks of data labeling handled by each candidate labeler and obtaining the net recommendation degree of data labeling of each candidate labeler according to the processing status of each historical task at each check node specifically includes: Obtain the time from now of all historical tasks of data annotation handled by each candidate annotator, and obtain the confidence weight of each historical task of each candidate annotator based on the time from now; Decompose each historical task according to each check node, determine the quality accuracy of the node feedback and the total number of node feedback of the decomposed historical task at each check node, and determine the equivalent accuracy of each historical task at each check node according to the quality accuracy and the total number of node feedback; Determine the constraint coefficient corresponding to each check node according to the equivalent accuracy rate, normalize the constraint coefficient, weight the normalized constraint coefficient and the equivalent accuracy rate to obtain the equivalent quality of each historical task, and obtain the confidence quality of each historical task according to the confidence weight and equivalent quality of each historical task; Determine the capacity efficiency of each historical task, and obtain the annotation quality of each historical task according to the confidence quality capacity efficiency of each historical task; The average of the annotation quality of all historical tasks handled by each candidate annotator is calculated to obtain the net recommendation of each candidate annotator. The optimization goal of the pre-scheduling is: in, represents the optimization target of the pre-scheduling; Indicates the total number of pre-scheduled labelers; Indicates the average response time of the pre-scheduled labelers; The constraints of the pre-scheduling are: in, Indicates The predicted capacity of the pre-scheduled labelers; Indicates the target capacity of the project to be delivered; Indicates The net recommendation of the pre-scheduled annotators; Indicates the preset value; The optimization goal of the dynamic scheduling is: in, represents the optimization target of the dynamic scheduling; Indicates the total number of dynamically scheduled annotation personnel; Indicates the average response time of dynamically scheduled labelers; The constraints of the dynamic scheduling are: in, Indicates The predicted capacity of the labeling personnel who are dynamically scheduled; Indicates The predicted capacity of the marking personnel in the current job batch; Indicates the total number of labelers in the current job batch; when ± is +, it means that more labelers need to be added, and when ± is -, it means that fewer labelers need to be added. In addition, when it is necessary to add labelers, the net recommendation degree of each dynamically scheduled labeler ≥ the preset value, and when it is necessary to reduce labelers, the net recommendation degree of the dynamically scheduled labelers ∄ ≤ the net recommendation degree of the unscheduled labelers ∀.

2. The data labeling method for autonomous driving data according to claim 1, characterized in that: The method of obtaining all historical tasks of data labeling handled by each candidate labeler and obtaining the net recommendation degree of data labeling of each candidate labeler according to the processing status of each historical task at each check node also includes: The historical tasks are screened to remove the historical tasks that have not gone through a complete inspection node; the inspection node includes a review node, a quality inspection node, and an acceptance node that are performed sequentially.

3. The data labeling method for autonomous driving data according to claim 1, characterized in that: Before the step of determining the response time of each candidate labeler, pre-scheduling the project to be delivered according to the predicted capacity and response time of the candidate labelers, and allocating labelers in the previous operation batch to the project to be delivered, the method further includes: Determine the scheduling degree of each candidate labeler within a preset time, and exclude candidate labelers whose scheduling times exceed the preset degree according to the scheduling degree; the scheduling degree is the scheduling times / job task volume of the current batch scheduling ratio of the candidate labeler, and the current batch scheduling ratio is the scheduling times / batch task volume within the preset time.

4. The data labeling method for autonomous driving data according to claim 1, characterized in that: The method for determining the response time specifically includes: The response time of the labeling personnel to be scheduled is determined according to the working status of the labeling personnel to be scheduled; the working status includes an idle state and a working state. The response time of the labeling personnel to be scheduled in the idle state is 0, and the response time of the labeling personnel to be scheduled in the working state is the remaining task output value / predicted capacity of the labeling personnel to be scheduled.

5. A data labeling device for autonomous driving data, characterized in that: The device comprises: The historical task processing module is used to obtain the historical tasks of all data annotations processed by each candidate annotator, and obtain the net recommendation degree of data annotation of each candidate annotator based on the processing status of each historical task at each check node; The project capacity prediction module is used to decompose historical tasks and tasks to be scheduled, predict the difficulty of tasks to be scheduled based on the processing status of historical tasks at each inspection node, and determine the target capacity of the project to be delivered based on the difficulty of tasks to be scheduled; A personnel capacity prediction module is used to obtain the predicted capacity of each candidate labeling personnel according to the net recommendation degree of each candidate labeling personnel and the task difficulty of the task to be scheduled; The project pre-scheduling module is used to determine the response time of each candidate labeler, pre-scheduling the projects to be delivered based on the predicted capacity and response time of the candidate labelers, and assigning labelers in the previous batch to the projects to be delivered; The project dynamic scheduling module is used to determine the actual completion status of the projects to be delivered. If the completion status does not meet expectations, the project to be delivered is dynamically scheduled based on the total predicted capacity of the labelers in the previous batch and the predicted capacity of the candidate labelers, and the labelers in the previous batch are adjusted for the projects to be delivered. The historical task processing module specifically includes: Obtain the time from now of all historical tasks of data annotation handled by each candidate annotator, and obtain the confidence weight of each historical task of each candidate annotator based on the time from now; Decompose each historical task according to each check node, determine the quality accuracy of the node feedback and the total number of node feedback of the decomposed historical task at each check node, and determine the equivalent accuracy of each historical task at each check node according to the quality accuracy and the total number of node feedback; Determine the constraint coefficient corresponding to each check node according to the equivalent accuracy rate, normalize the constraint coefficient, weight the normalized constraint coefficient and the equivalent accuracy rate to obtain the equivalent quality of each historical task, and obtain the confidence quality of each historical task according to the confidence weight and equivalent quality of each historical task; Determine the capacity efficiency of each historical task, and obtain the annotation quality of each historical task according to the confidence quality capacity efficiency of each historical task; The average of the annotation quality of all historical tasks handled by each candidate annotator is calculated to obtain the net recommendation of each candidate annotator. The optimization goal of the pre-scheduling is: in, represents the optimization target of the pre-scheduling; Indicates the total number of pre-scheduled labelers; Indicates the average response time of the pre-scheduled labelers; The constraints of the pre-scheduling are: in, Indicates The predicted capacity of the pre-scheduled labelers; Indicates the target capacity of the project to be delivered; Indicates The net recommendation of the pre-scheduled annotators; Indicates the preset value; The optimization goal of the dynamic scheduling is: in, represents the optimization target of the dynamic scheduling; Indicates the total number of dynamically scheduled annotation personnel; Indicates the average response time of dynamically scheduled labelers; The constraints of the dynamic scheduling are: in, Indicates The predicted capacity of the labeling personnel who are dynamically scheduled; Indicates The predicted capacity of the marking personnel in the current job batch; Indicates the total number of labelers in the current job batch; when ± is +, it means that more labelers need to be added, and when ± is -, it means that fewer labelers need to be added. In addition, when it is necessary to add labelers, the net recommendation degree of each dynamically scheduled labeler ≥ the preset value, and when it is necessary to reduce labelers, the net recommendation degree of the dynamically scheduled labelers ∄ ≤ the net recommendation degree of the unscheduled labelers ∀.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the data labeling method for autonomous driving data as described in any one of claims 1 to 4 are implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data labeling method for autonomous driving data as described in any one of claims 1 to 4 are implemented.

Images