Annotated data multiplexing method, system, device and computer readable storage medium
By comparing the dataset identifiers of the annotation task and the training task, the reasons for the unused annotation data are analyzed, and processing instructions are generated, which solves the problem of the ineffective use of annotation data and realizes efficient management and reuse of resources.
Patent Information
- Application Number
- CN202611287473.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-24
- Publication Date
- 2026-11-06
AI Technical Summary
In existing technologies, labeled data lacks correlation between delivery and training, making it difficult to accurately identify labeled data that is not effectively utilized by the training task, resulting in resource waste and low management efficiency.
By comparing the identifiers of the labeled task dataset and the training task dataset, unused labeled data and their target identifiers are identified. Combined with the target annotation specification version and scenario, the categories of reasons for non-use are analyzed, and corresponding data processing instructions are generated to achieve accurate identification and effective processing of unused labeled data.
It enables accurate identification and effective processing of unused labeled data, avoids waste of storage resources, and significantly improves data management efficiency.
Smart Images
Figure CN123048522A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, specifically to a method, system, device, and computer-readable storage medium for reusing labeled data. Background Technology
[0002] With the continuous development of artificial intelligence technology, model training is becoming increasingly reliant on labeled data. In fields such as autonomous driving and computer vision, data labeling is costly, and as model iteration speeds accelerate and labeling standards change, efficiently reusing historical labeled data and avoiding data asset idleness are crucial for reducing R&D costs.
[0003] In related technologies, the management of labeled data typically focuses on quality control before data delivery and maintenance of storage status after delivery. After data delivery, labeled data is stored in a data pool, and training tasks select data from the data pool for loading according to needs. Data that is not loaded is often directly deleted or left unused for a long time. However, in existing technologies, there is a lack of connection between the delivery and training phases, making it difficult to accurately identify labeled data that is not effectively used by training tasks. Furthermore, for labeled data that is not effectively used by training tasks, there is a lack of effective decision-making basis to determine its subsequent processing direction, resulting in some labeled data not being effectively reused or properly archived, leading to a waste of storage resources and low data management efficiency.
[0004] Therefore, how to provide a method for reusing labeled data to achieve accurate identification and processing of unused labeled data is an urgent problem to be solved. Summary of the Invention
[0005] This application provides a method, system, device, and computer-readable storage medium for reusing labeled data, which can achieve accurate identification and processing of unused labeled data.
[0006] In a first aspect, embodiments of this application provide a method for reusing annotation data, the method comprising: The unused labeled data and their corresponding target labels are determined based on the first identifier corresponding to the labeled task dataset and the second identifier corresponding to the training task dataset. Based on the target identifier, determine the target annotation specification version and target scenario corresponding to the unused annotation data; Determine the category of reasons for the non-use of unused annotation data based on the current training task, the version of the target annotation specification, and the target scenario; Generate data processing instructions corresponding to the unused labeled data based on the category of reasons for unuse.
[0007] In conjunction with the first aspect, in one implementation, determining the category of unused reasons for unused labeled data based on the current training task, the target annotation specification version, and the target scenario includes: The version specification is determined based on the target annotation specification version and the annotation version corresponding to the current training task; Scene redundancy is determined based on the accuracy of the current training task in the target scene, the accuracy of the historical training task in the target scene, and the cumulative number of occurrences of the target scene in the historical training set. Determine the model convergence based on the training task data corresponding to the current training task; The categories of reasons for the non-use of unused labeled data are determined based on version standardization, scenario redundancy, and model convergence.
[0008] In conjunction with the first aspect, in one implementation, determining the version specification based on the target annotation specification version and the annotation version corresponding to the current training task includes: If the target annotation specification version is inconsistent with the annotation version corresponding to the current training task, it is judged as non-standard version; If the target annotation specification version is consistent with the annotation version corresponding to the current training task, it is determined to be a version specification.
[0009] In conjunction with the first aspect, in one implementation, determining scene redundancy based on the accuracy of the current training task in the target scene, the accuracy of historical training tasks in the target scene, and the cumulative occurrence count of the target scene in the historical training set includes: The performance gain of the model in the target scene is determined based on the accuracy of the current training task in the target scene and the accuracy of the historical training tasks in the target scene. If the cumulative occurrence count of the target scene in the historical training set is greater than the preset count threshold and the performance gain rate is less than the preset gain rate threshold, then it is determined to be a scene redundancy. If the cumulative occurrence count of the target scene in the historical training set is not greater than the preset count threshold or the performance gain rate is not less than the preset gain rate threshold, then the scene is determined to be non-redundant.
[0010] In conjunction with the first aspect, in one implementation, determining the model convergence based on the training task data corresponding to the current training task includes: Determine the model loss value based on the training task data corresponding to the current training task; If the model loss value is not less than the preset loss threshold, the model is determined to be non-convergent. If the model loss value is less than the preset loss threshold, the model is considered to have converged.
[0011] In conjunction with the first aspect, in one implementation, generating data processing instructions corresponding to unused labeled data based on the unused reason category includes: If the reason for non-use is version non-standard, a re-labeling instruction is generated for the original sensor data corresponding to the unused labeled data; If the reason for not using the data is scenario redundancy or model convergence, a data archiving instruction for the unused labeled data will be generated.
[0012] Secondly, embodiments of this application provide a labeled data reuse system, the labeled data reuse system comprising: The first processing module is used to determine unused labeled data and their corresponding target labels based on the first identifier corresponding to the labeled task dataset and the second identifier corresponding to the training task dataset. The second processing module is used to determine the target annotation specification version and target scenario corresponding to the unused annotation data based on the target identifier; The third processing module is used to determine the category of reasons for the unused unused labeled data based on the current training task, the version of the target annotation specification, and the target scenario. The fourth processing module is used to generate data processing instructions corresponding to the unused labeled data based on the category of unused reasons.
[0013] In conjunction with the second aspect, in one implementation, the third processing module is specifically used for: The version specification is determined based on the target annotation specification version and the annotation version corresponding to the current training task; Scene redundancy is determined based on the accuracy of the current training task in the target scene, the accuracy of the historical training task in the target scene, and the cumulative number of occurrences of the target scene in the historical training set. Determine the model convergence based on the training task data corresponding to the current training task; The categories of reasons for the non-use of unused labeled data are determined based on version standardization, scenario redundancy, and model convergence.
[0014] Thirdly, embodiments of this application provide a labeling data reuse device, the labeling data reuse device including a processor, a memory, and a labeling data reuse program stored in the memory and executable by the processor, wherein when the labeling data reuse program is executed by the processor, it implements the steps of the labeling data reuse method as described in any of the preceding claims.
[0015] Fourthly, embodiments of this application provide a computer-readable storage medium storing a labeling data reuse program, wherein when the labeling data reuse program is executed by a processor, it implements the steps of the labeling data reuse method as described in any of the preceding claims.
[0016] The beneficial effects of the technical solutions provided in this application include: By comparing the first identifier corresponding to the annotation task dataset with the second identifier corresponding to the training task dataset, unused labeled data and their corresponding target identifiers are identified. This identifier comparison enables accurate identification of unused labeled data. Then, based on the target identifiers, the target annotation specification version and target scenario corresponding to the unused labeled data are determined. Furthermore, based on the current training task, target annotation specification version, and target scenario, the category of unused reasons for the unused labeled data is determined. By combining data attributes with current training requirements, a clear decision-making basis for handling idle data is provided. Finally, based on the category of unused reasons, data processing instructions corresponding to the unused labeled data are generated, enabling the system to take appropriate processing measures according to the specific reasons for unuse. This achieves effective reuse or reasonable archiving of unused labeled data, avoids waste of storage resources, and significantly improves data management efficiency. Attached Figure Description
[0017] Figure 1 This is a flowchart illustrating an embodiment of the data reuse method described in this application. Figure 2 A detailed flowchart illustrating an embodiment of the data reuse method described in this application; Figure 3 This is a schematic diagram of the functional modules of an embodiment of the data reuse system for annotation in this application; Figure 4 This is a schematic diagram of the hardware structure of the annotation data reuse device involved in the embodiments of this application. Detailed Implementation
[0018] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.
[0019] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.
[0020] In a first aspect, embodiments of this application provide a method for reusing labeled data.
[0021] In one embodiment, reference is made to Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the data reuse method described in this application. Figure 1 As shown, the methods for reusing labeled data include: Step S10: Determine the unused labeled data and their corresponding target labels based on the first identifier corresponding to the labeled task dataset and the second identifier corresponding to the training task dataset.
[0022] In an exemplary embodiment of this application, the system first establishes a delivery log during the annotation delivery phase to record the first identifier and its associated information. When the annotation platform completes the annotation of a batch of data and delivers it, the system generates a globally unique first identifier (e.g., using UUID format) for that batch of data and writes the first identifier and delivery information into the delivery log. The delivery information includes, but is not limited to: the corresponding list of original sensor data IDs, annotation task type, annotation specification version number, scene type label, quality score, and delivery timestamp. Simultaneously, the system stores the annotation files in distributed object storage and binds the storage path to the first identifier. At this point, the status of the batch of data is marked as "delivered," indicating that the data has entered the data asset management system but has not yet been loaded by any training task.
[0023] Secondly, during the training set loading phase, the system establishes a training log to record the second identifier and its associated information. When the model training system starts a training task, it assigns a unique task ID to the task and reads the list of datasets to be loaded from the configuration file. At the start of training, the system writes the second identifier (i.e., dataset ID), task ID, start timestamp, model version number, and the number of epochs traversed for each dataset to the training log for all datasets used in this task. Through the unique identifier of the dataset, the system can associate the first identifier in the delivery log with the second identifier in the training log; if the second identifier of a dataset appears in the training log, the status of that data changes from "delivered" to "used".
[0024] Finally, the system determines unused labeled data based on the aforementioned log records and data status. The system defines several data states and automatically converts them according to rules, including: delivered, used, unused (timeout), and archived (not for training purposes). To identify unused labeled data, the system periodically or upon triggering an event extracts all first identifiers from the delivery log to construct a delivery identifier set, and extracts all second identifiers from the training log within a preset time window to construct a training identifier set. The system compares these two sets (e.g., calculates the difference) to identify identifiers that exist in the delivery identifier set but not in the training identifier set. Further, the system combines the delivery timestamp for judgment. If the data delivery time corresponding to a certain first identifier exceeds a preset threshold (e.g., 30 days) and there is still no loading record, the status of the data is updated to "unused (timeout)," and the first identifier is identified as the target identifier; its corresponding dataset is the unused labeled data. Through this mechanism based on identifier comparison and status tracking, the system can accurately filter out delivered but not actually loaded data from massive amounts of data, providing clear target objects for subsequent processing.
[0025] Step S20: Determine the target annotation specification version and target scenario corresponding to the unused annotation data based on the target identifier.
[0026] In this exemplary embodiment, the system pre-establishes a data asset management database (or metadata database) that stores detailed attribute information of all delivered datasets, using the dataset identifier as the primary key index. After determining the target identifier, the system uses the target identifier to query the data asset management database, parsing the key attribute fields associated with the target identifier from the query results. These include fields recording the rules followed during data annotation (i.e., the target annotation specification version) and fields recording the characteristics of the data acquisition environment (i.e., the target scene). For example, the system reads the spec_version field from the metadata to obtain the target annotation specification version (e.g., V1.0) and reads the scene_type field to obtain the target scene (e.g., a rainy highway). In addition, the system can also simultaneously read other auxiliary information associated with the target identifier, such as the original data storage path and delivery time. Through this identifier-based metadata retrieval mechanism, the system can quickly obtain the static attributes of unused annotated data, providing a data foundation for analyzing the reasons for its unuse.
[0027] Step S30: Determine the category of reasons for the unused unused annotation data based on the current training task, the version of the target annotation specification, and the target scenario.
[0028] In this exemplary embodiment, the system first acquires the requirements of the current training task, such as the annotation specification version required for the current model training, the performance status of the current model in different scenarios, and the scenario distribution of the current training set. The system then performs a compatibility comparison between the target annotation specification version and the annotation version required for the current training task to analyze whether there are any specification conflicts. Simultaneously, it performs a matching analysis between the target scenario and the scenario distribution and model performance status required for the current training task to assess whether the scenario data is redundant or whether the model has converged. Based on the above analysis results, the system categorizes unused annotation data into specific unused reason categories. Through this multi-dimensional matching analysis, the system can clearly identify the root cause of data unuse, thereby providing a decision-making basis for subsequently generating targeted processing instructions.
[0029] Step S40: Generate data processing instructions corresponding to the unused labeled data based on the unused reason category.
[0030] In this embodiment, as an example, the system determines the corresponding data processing strategy based on the category of unused reasons. If the unused reason category is version non-standard, a re-labeling instruction is generated. This re-labeling instruction includes the path information of the original sensor data associated with the unused labeled data and the new labeling standard version, and dynamically sets the task priority according to the scarcity of the scene. If the unused reason category is scene redundancy or model convergence, a data archiving instruction is generated. This data archiving instruction includes data status change information and target use allocation information, which is used to allocate the data to the test set, simulation library, or difficult example library, and remove the data from the active index of the training candidate pool. By establishing a mapping relationship between the unused reason category and the data processing instruction, differentiated processing of data assets is achieved.
[0031] This application identifies unused labeled data and their corresponding target identifiers by comparing a first identifier corresponding to the labeled task dataset with a second identifier corresponding to the training task dataset. This identifier comparison enables accurate identification of unused labeled data. Furthermore, based on the target identifiers, the application determines the target annotation specification version and target scenario corresponding to the unused labeled data. Then, based on the current training task, target annotation specification version, and target scenario, the application determines the category of unused reasons for the unused labeled data. By combining data attributes with current training requirements, the application provides a clear decision-making basis for handling idle data. Finally, based on the category of unused reasons, the application generates data processing instructions corresponding to the unused labeled data, enabling the system to take appropriate processing measures according to the specific reasons for unuse. This achieves effective reuse or reasonable archiving of unused labeled data, avoids waste of storage resources, and significantly improves data management efficiency.
[0032] Further, in one embodiment, determining the category of unused reasons for unused labeled data based on the current training task, the target annotation specification version, and the target scene includes: The version specification is determined based on the target annotation specification version and the annotation version corresponding to the current training task; Scene redundancy is determined based on the accuracy of the current training task in the target scene, the accuracy of the historical training task in the target scene, and the cumulative number of occurrences of the target scene in the historical training set. Determine the model convergence based on the training task data corresponding to the current training task; The categories of reasons for the non-use of unused labeled data are determined based on version standardization, scenario redundancy, and model convergence.
[0033] As an example, in this embodiment, the metadata of the unused labeled data is first read to obtain the target label specification version at the time of delivery, and the label version corresponding to the current training task is also read. Secondly, historical training records are queried to obtain the accuracy of the current version (the model version corresponding to the current training task) on the target scene, the accuracy of historical training tasks on the target scene, and the cumulative number of occurrences of the target scene in the historical training set. Thirdly, the training task data corresponding to the current training task is obtained. Finally, the category of unused reasons for the unused labeled data is determined based on the evaluation results of the above three dimensions.
[0034] The specific mapping logic is as follows: If the version standardization assessment result is "version non-standard," then the reason for unuse is prioritized as standard incompatibility (version non-standard), because inconsistent standards prevent data from being directly input into the model; if the version standardization assessment result is "version standard," but the scenario redundancy assessment result is "scenario redundancy," then the reason for unuse is determined as "scenario redundancy"; if the version standardization assessment result is "version standard" and the scenario redundancy assessment result is "scenario non-redundancy," but the model convergence assessment result is "model convergence," then the reason for unuse is determined as "model convergence." It should be noted that if none of the above three conditions are met, then the reason for unuse is determined as "other," and is submitted for manual review. Through this multi-dimensional comprehensive judgment, the system can accurately distinguish whether data is unused because it is "unusable" (standardization issue), "unnecessary" (scenario redundancy), or "unnecessary to use" (model convergence), laying the foundation for generating differentiated instructions subsequently.
[0035] Furthermore, in one embodiment, determining the version specification based on the target annotation specification version and the annotation version corresponding to the current training task includes: If the target annotation specification version is inconsistent with the annotation version corresponding to the current training task, it is judged as non-standard version; If the target annotation specification version is consistent with the annotation version corresponding to the current training task, it is determined to be a version specification.
[0036] As an example, in this embodiment, the target annotation specification version is compared with the annotation version corresponding to the current training task. If the target annotation specification version is inconsistent with the annotation version corresponding to the current training task (e.g., different version numbers or incompatible field definitions), it is determined to be a non-standard version; if the target annotation specification version is consistent with the annotation version corresponding to the current training task, it is determined to be a standard version. Furthermore, version inconsistency includes not only different version numbers but also changes in key field definitions. For example, the old version only includes 2D bounding box coordinates, while the new version requires 3D bounding box coordinates and attribute fields. In this case, the system can also determine that the version is non-standard by verifying the schema field. This embodiment ensures, through strict version comparison, that only data conforming to the current specification is excluded from the "non-standard version" category.
[0037] Further, in one embodiment, determining scene redundancy based on the accuracy of the current training task in the target scene, the accuracy of historical training tasks in the target scene, and the cumulative occurrence count of the target scene in the historical training set includes: The performance gain of the model in the target scene is determined based on the accuracy of the current training task in the target scene and the accuracy of the historical training tasks in the target scene. If the cumulative occurrence count of the target scene in the historical training set is greater than the preset count threshold and the performance gain rate is less than the preset gain rate threshold, then it is determined to be a scene redundancy. If the cumulative occurrence count of the target scene in the historical training set is not greater than the preset count threshold or the performance gain rate is not less than the preset gain rate threshold, then the scene is determined to be non-redundant.
[0038] As an example, in this embodiment, scene redundancy is used to determine whether a scene to which unlabeled data belongs has been sufficiently learned in historical training, avoiding resource waste caused by repeated training; the specific calculation process is as follows: The system first queries the database to obtain the accuracy of the current version on the target scene and the accuracy of the historical version on the target scene, and determines the performance gain rate of the model on the target scene based on the accuracy of the current training task on the target scene and the accuracy of the historical training task on the target scene. The calculation formula is:
[0039] In the formula, This represents the accuracy of the current training task in the target scenario. The accuracy of historical training tasks in the target scenario; This represents the performance gain of the model in the target scene. It must be a very small positive number to prevent division by zero, and it must satisfy... .
[0040] It should be noted that the specific values of the preset frequency threshold and the preset gain rate threshold can be determined according to actual needs and are not limited here. For example, the preset frequency threshold can be preferably set to 5000, and the preset gain rate threshold can be preferably set to 0.01. The calculation results are compared with the preset thresholds: if the cumulative occurrence of the target scene in the historical training set is greater than the preset frequency threshold and the performance gain rate is less than the preset gain rate threshold, it indicates that the data volume of the scene is sufficient and the improvement of the new model in the scene is limited, and it is judged as scene redundancy; if the cumulative occurrence of the target scene in the historical training set is not greater than the preset frequency threshold or the performance gain rate is not less than the preset gain rate threshold, it indicates that the data volume of the scene is not yet sufficient or the new model has great potential for improvement in the scene, and it is judged as scene non-redundancy. For example, a certain "crossroads-daytime" scene has appeared 6200 times, which is greater than 5000 times. The accuracy of the previous version was 94.2%, the current version is 94.3%, and the gain rate is only 0.001, which is less than 0.01, so it is judged as scene redundancy. This embodiment avoids the subjectivity of human experience judgment by using quantitative indicators and accurately identifies low-value duplicate data.
[0041] Further, in one embodiment, determining the model convergence based on the training task data corresponding to the current training task includes: Determine the model loss value based on the training task data corresponding to the current training task; If the model loss value is not less than the preset loss threshold, the model is determined to be non-convergent. If the model loss value is less than the preset loss threshold, the model is considered to have converged.
[0042] In this exemplary embodiment, the system obtains the current version of the model from the training task data corresponding to the current training task, uses the current version of the model to infer the training task data to obtain the prediction result, and selects the corresponding loss function according to the task type to calculate the difference between the prediction result and the label corresponding to the training task data as the model loss value Loss(d). For example, cross-entropy or Smooth L1 loss is used for perception tasks, and classification loss plus regression loss is used for detection tasks. Specifically, the specific value of the preset loss threshold can be determined according to actual needs and is not limited here. For example, the preset loss threshold can preferably be 0.1. The system compares the model loss value with the preset loss threshold: if the model loss value is less than the preset loss threshold, it indicates that the prediction error of the model in this scenario has been reduced to a low level, and further training is unlikely to significantly improve performance, so it is determined that the model has converged; if the model loss value is not less than the preset loss threshold, it indicates that the model has not yet learned the scenario data well, so it is determined that the model has not converged. This embodiment effectively identifies the data idle situation caused by model capacity saturation through the objective indicator of loss value.
[0043] It should be noted that the thresholds mentioned in the above embodiments are not fixed. After each model iteration cycle, the system can automatically adjust the thresholds using Bayesian optimization based on the correlation between historical judgment results and actual model performance improvement. This process is common knowledge in the field and will not be elaborated here for the sake of simplicity. For example, if the initial convergence loss threshold is set to 0.1, and after two runs it is found that many data points judged as "model converged" can actually improve performance in the later version of the model, the system will automatically adjust the threshold down to 0.08 to reduce false judgments.
[0044] Furthermore, in one embodiment, the step of generating data processing instructions corresponding to the unused labeled data based on the unused reason category includes: If the reason for non-use is version non-standard, a re-labeling instruction is generated for the original sensor data corresponding to the unused labeled data; If the reason for not using the data is scenario redundancy or model convergence, a data archiving instruction for the unused labeled data will be generated.
[0045] As an example, in this embodiment of the application, the system generates executable processing instructions based on the unused reason category determined in the aforementioned steps, so as to achieve optimized disposal of data assets. The specific generation logic is as follows: (1) Re-labeling instruction generation: When the reason for unuse is version non-standard, the system determines that the old labeling is invalid but the original data is still valuable, and generates a re-labeling instruction for the original sensor data corresponding to the unused labeled data. Specifically, the system obtains the storage path list of the original sensor data from the delivery log. The original sensor data includes camera image sequences, LiDAR point clouds, millimeter-wave radar target lists, and CAN bus signals, and verifies that the files exist and are complete; then it calls the API of the labeling platform to create a new labeling task. The task name is automatically generated as "re-labeling_" plus the original dataset name plus a timestamp. The labeling specification version is set to the version required by the current model. The input data ID is the list of obtained original data IDs, and the task priority is dynamically calculated according to the scarcity of the scene. ,in, The system prioritizes tasks based on the frequency of scene occurrences, automatically placing higher-priority tasks at the top. After a new task is successfully created, its status is recorded in the task management table, and the status of the original data is updated to "under re-marking" to prevent duplicate triggering. This operation reuses the original sensor data, avoiding the high cost of re-collection.
[0046] (2) Data archiving instruction generation: When the reason for unuse is scene redundancy or model convergence, the system determines that the labeled data itself is valid but does not need to be trained at present, and generates a data archiving instruction for the unused labeled data. Specifically, the system changes the data status from "delivered" to "archived (non-training purpose)" and records the reason and time of the change; it automatically assigns appropriate uses according to the data characteristics, adds data with quality scores higher than a preset threshold (e.g., 0.9) to the test set for accuracy verification before the model version is released, converts data containing dynamic interactive events into simulation scenarios and injects them into the simulation platform for closed-loop testing, and adds data with loss values at a medium level to the difficult example analysis library for algorithm engineers to perform targeted optimization; at the same time, it removes the data from the active index of the "training candidate pool" and marks it as "low priority retrieval" to reduce the computational overhead of daily scanning, but retains the complete asset without deletion. This operation preserves the data asset for future testing or iteration use, while optimizing storage resources.
[0047] It should be understood that for data tagged "Other," the system generates a manual review work order and pushes it to the data management platform's operation panel. Limited by the annotation platform's manual review resources (maximum of 200 orders per day), the system sorts the data in descending order of scenario scarcity, reviewing high-value data first. The manual reviewer can choose to: force relabeling, archive to the test set, or permanently discard; the system records the manual decision, which will later serve as positive and negative samples for rule engine optimization. Through the generation of the aforementioned differentiated instructions, this embodiment achieves refined processing of unused labeled data, both rescuing reusable original data and rationally managing idle labeled assets.
[0048] It should be noted that after creating a new annotation task, the system waits for the annotation platform to complete the task and notify it through a callback interface. After receiving the new annotation results, a new dataset ID is generated and written to the delivery log in the manner of the first stage. At the same time, the original sensor data is retained in the original data ID field, and the original dataset ID is recorded in the parent dataset ID field to establish data lineage. The status of the newly annotated data is set to "delivered" and waits for the training system to load it. The system continues to track whether it is actually used through the training log, thus completing the closed loop of "delivery → unused → re-labeling → re-delivery → re-use". After each closed loop is completed, the system calculates the re-labeling efficiency (the number of data loaded for training after re-labeling divided by the total number of data that triggers re-labeling) and the cost saving rate (1 - re-labeling cost divided by the estimated cost of re-collecting and labeling equivalent data). Then, based on these two indicators, Bayesian optimization is used to adjust the threshold parameters in the second-stage rule engine, with the goal of maximizing the re-labeling efficiency. In addition, considering the practical constraints of limited data collection vehicles, for high-priority supplementary data collection needs, the system will assess the remaining data collection resources, break down large tasks into multiple sub-batch and provide the estimated completion time. Through the aforementioned closed-loop execution and parameter optimization, the system can continuously iterate the judgment rules, thereby constantly improving the accuracy and efficiency of data reuse.
[0049] It should be understood that, with reference Figure 2 As shown, starting from the delivery of labeled data, a unique ID is generated upon delivery, and training loading records are completed to establish the correspondence between task ID and dataset ID. Subsequently, the system periodically scans and identifies unused data, which is then handed over to the rule engine to classify the reasons. The rule engine divides unused data into different processing directions, such as non-standard version, redundant scenario, or model convergence. Among them, the non-standard version branch will first locate the original data, create a new labeling task according to the new specification, and complete the delivery of new labeling results. The redundant scenario or model convergence branch will change the data status to "archived" and allocate it to the test set, simulation library, or difficult example library for archiving and reuse. The new data produced by the relabeling branch will enter the tracking stage to fully record the data lineage. After the archiving and reuse stage is completed, the active index will be released while the storage assets are retained. All paths will eventually be summarized in the effect evaluation stage, and indicators will be calculated from two dimensions: efficiency and saving rate. Bayesian optimization will be used to adjust the threshold parameters of the rule engine to form a complete management closed loop.
[0050] Secondly, embodiments of this application also provide a system for reusing labeled data.
[0051] In one embodiment, reference is made to Figure 3 , Figure 3 This is a schematic diagram of the functional modules of an embodiment of the data reuse system for annotation in this application. Figure 3 As shown, the labeled data reuse system includes: The first processing module is used to determine unused labeled data and their corresponding target labels based on the first identifier corresponding to the labeled task dataset and the second identifier corresponding to the training task dataset. The second processing module is used to determine the target annotation specification version and target scenario corresponding to the unused annotation data based on the target identifier; The third processing module is used to determine the category of reasons for the unused unused labeled data based on the current training task, the version of the target annotation specification, and the target scenario. The fourth processing module is used to generate data processing instructions corresponding to the unused labeled data based on the category of unused reasons.
[0052] Furthermore, in one embodiment, the third processing module is specifically used for: The version specification is determined based on the target annotation specification version and the annotation version corresponding to the current training task; Scene redundancy is determined based on the accuracy of the current training task in the target scene, the accuracy of the historical training task in the target scene, and the cumulative number of occurrences of the target scene in the historical training set. Determine the model convergence based on the training task data corresponding to the current training task; The categories of reasons for the non-use of unused labeled data are determined based on version standardization, scenario redundancy, and model convergence.
[0053] Furthermore, in one embodiment, determining the version specification based on the target annotation specification version and the annotation version corresponding to the current training task includes: If the target annotation specification version is inconsistent with the annotation version corresponding to the current training task, it is judged as non-standard version; If the target annotation specification version is consistent with the annotation version corresponding to the current training task, it is determined to be a version specification.
[0054] Furthermore, in one embodiment, the third processing module is specifically used for: The performance gain of the model in the target scene is determined based on the accuracy of the current training task in the target scene and the accuracy of the historical training tasks in the target scene. If the cumulative occurrence count of the target scene in the historical training set is greater than the preset count threshold and the performance gain rate is less than the preset gain rate threshold, then it is determined to be a scene redundancy. If the cumulative occurrence count of the target scene in the historical training set is not greater than the preset count threshold or the performance gain rate is not less than the preset gain rate threshold, then the scene is determined to be non-redundant.
[0055] Furthermore, in one embodiment, the third processing module is specifically used for: Determine the model loss value based on the training task data corresponding to the current training task; If the model loss value is not less than the preset loss threshold, the model is determined to be non-convergent. If the model loss value is less than the preset loss threshold, the model is considered to have converged.
[0056] Furthermore, in one embodiment, the fourth processing module is specifically used for: If the reason for non-use is version non-standard, a re-labeling instruction is generated for the original sensor data corresponding to the unused labeled data; If the reason for not using the data is scenario redundancy or model convergence, a data archiving instruction for the unused labeled data will be generated.
[0057] This application identifies unused labeled data and their corresponding target identifiers by comparing a first identifier corresponding to the labeled task dataset with a second identifier corresponding to the training task dataset. This identifier comparison enables accurate identification of unused labeled data. Furthermore, based on the target identifiers, the application determines the target annotation specification version and target scenario corresponding to the unused labeled data. Then, based on the current training task, target annotation specification version, and target scenario, the application determines the category of unused reasons for the unused labeled data. By combining data attributes with current training requirements, the application provides a clear decision-making basis for handling idle data. Finally, based on the category of unused reasons, the application generates data processing instructions corresponding to the unused labeled data, enabling the system to take appropriate processing measures according to the specific reasons for unuse. This achieves effective reuse or reasonable archiving of unused labeled data, avoids waste of storage resources, and significantly improves data management efficiency.
[0058] The functions of each module in the above-mentioned annotation data reuse system correspond to the steps in the above-mentioned annotation data reuse method embodiment, and their functions and implementation processes will not be described in detail here.
[0059] Thirdly, embodiments of this application provide a labeled data reuse device, which can be a personal computer (PC), laptop computer, server, or other device with data processing capabilities.
[0060] Reference Figure 4 , Figure 4 This is a schematic diagram of the hardware structure of the annotation data multiplexing device involved in the embodiments of this application. In the embodiments of this application, the annotation data multiplexing device may include a processor, a memory, a communication interface, and a communication bus.
[0061] The communication bus can be of any type and is used to interconnect the processor, memory, and communication interface.
[0062] Communication interfaces include input / output (I / O) interfaces, physical interfaces, and logical interfaces used for interconnecting devices within the annotation data multiplexing device, as well as interfaces used for interconnecting the annotation data multiplexing device with other devices (such as other computing devices or user equipment). Physical interfaces can be Ethernet interfaces, fiber optic interfaces, ATM interfaces, etc.; user equipment can be displays, keyboards, etc.
[0063] Memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.
[0064] The processor can be a general-purpose processor, which can call the annotation data reuse program stored in memory and execute the annotation data reuse method provided in the embodiments of this application. For example, the general-purpose processor can be a central processing unit (CPU). The method executed when the annotation data reuse program is called can be referred to in the various embodiments of the annotation data reuse method of this application, and will not be repeated here.
[0065] Those skilled in the art will understand that Figure 4 The hardware structure shown does not constitute a limitation of this application and may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0066] Fourthly, embodiments of this application also provide a readable storage medium.
[0067] The present application has a readable storage medium storing a labeling data reuse program, wherein when the labeling data reuse program is executed by a processor, it implements the steps of the labeling data reuse method as described above.
[0068] The method implemented when the annotation data reuse procedure is executed can be referred to in various embodiments of the annotation data reuse method of this application, and will not be repeated here.
[0069] The terms "comprising" and "having," and any variations thereof, in the specification and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.
[0070] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.
[0071] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.
[0072] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.
[0073] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0074] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods described in the various embodiments of this application.
[0075] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.
Claims
1. A method for reusing labeled data, characterized in that, The method for reusing labeled data includes: The unused labeled data and their corresponding target labels are determined based on the first identifier corresponding to the labeled task dataset and the second identifier corresponding to the training task dataset. Based on the target identifier, determine the target annotation specification version and target scenario corresponding to the unused annotation data; Determine the category of reasons for the non-use of unused annotation data based on the current training task, the version of the target annotation specification, and the target scenario; Generate data processing instructions corresponding to the unused labeled data based on the category of reasons for unuse.
2. The annotation data reuse method as described in claim 1, characterized in that, The method for determining the categories of reasons for the unused unused labeled data based on the current training task, the version of the target annotation specification, and the target scenario includes: The version specification is determined based on the target annotation specification version and the annotation version corresponding to the current training task; Scene redundancy is determined based on the accuracy of the current training task on the target scene, the accuracy of the historical training task on the target scene, and the cumulative number of occurrences of the target scene in the historical training set. Determine the model convergence based on the training task data corresponding to the current training task; The categories of reasons for the non-use of unused labeled data are determined based on version standardization, scenario redundancy, and model convergence.
3. The annotation data reuse method as described in claim 2, characterized in that, The process of determining version specification based on the target annotation specification version and the annotation version corresponding to the current training task includes: If the target annotation specification version is inconsistent with the annotation version corresponding to the current training task, it is judged as non-standard version; If the target annotation specification version is consistent with the annotation version corresponding to the current training task, it is determined to be a version specification.
4. The annotation data reuse method as described in claim 2, characterized in that, The determination of scene redundancy based on the accuracy of the current training task in the target scene, the accuracy of the historical training tasks in the target scene, and the cumulative occurrence count of the target scene in the historical training set includes: The performance gain of the model in the target scene is determined based on the accuracy of the current training task in the target scene and the accuracy of the historical training tasks in the target scene. If the cumulative occurrence count of the target scene in the historical training set is greater than the preset count threshold and the performance gain rate is less than the preset gain rate threshold, then it is determined to be a scene redundancy. If the cumulative occurrence count of the target scene in the historical training set is not greater than the preset count threshold or the performance gain rate is not less than the preset gain rate threshold, then the scene is determined to be non-redundant.
5. The annotation data reuse method as described in claim 2, characterized in that, The step of determining model convergence based on the training task data corresponding to the current training task includes: Determine the model loss value based on the training task data corresponding to the current training task; If the model loss value is not less than the preset loss threshold, the model is determined to be non-convergent. If the model loss value is less than the preset loss threshold, the model is considered to have converged.
6. The annotation data reuse method as described in claim 2, characterized in that, The generation of data processing instructions corresponding to the unused labeled data based on the unused reason category includes: If the reason for non-use is version non-standard, a re-labeling instruction is generated for the original sensor data corresponding to the unused labeled data; If the reason for not using the data is scenario redundancy or model convergence, a data archiving instruction for the unused labeled data will be generated.
7. A system for reusing labeled data, characterized in that, The labeled data reuse system includes: The first processing module is used to determine unused labeled data and their corresponding target labels based on the first identifier corresponding to the labeled task dataset and the second identifier corresponding to the training task dataset. The second processing module is used to determine the target annotation specification version and target scenario corresponding to the unused annotation data based on the target identifier; The third processing module is used to determine the category of reasons for the unused unused annotation data based on the current training task, the version of the target annotation specification, and the target scenario. The fourth processing module is used to generate data processing instructions corresponding to the unused labeled data based on the category of unused reasons.
8. The labeled data reuse system as described in claim 7, characterized in that, The third processing module is specifically used for: The version specification is determined based on the target annotation specification version and the annotation version corresponding to the current training task; Scene redundancy is determined based on the accuracy of the current training task on the target scene, the accuracy of the historical training task on the target scene, and the cumulative number of occurrences of the target scene in the historical training set. Determine the model convergence based on the training task data corresponding to the current training task; The categories of reasons for the non-use of unused labeled data are determined based on version standardization, scenario redundancy, and model convergence.
9. A labeled data reuse device, characterized in that, The annotation data multiplexing device includes a processor, a memory, and an annotation data multiplexing program stored in the memory and executable by the processor, wherein when the annotation data multiplexing program is executed by the processor, it implements the steps of the annotation data multiplexing method as described in any one of claims 1 to 6.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a labeling data reuse program, wherein when the labeling data reuse program is executed by a processor, it implements the steps of the labeling data reuse method as described in any one of claims 1 to 6.