A data processing method and apparatus

Abstract

Embodiments of the present application disclose a data processing method and device, the method comprising: based on a requirement description of a data processing task, obtaining a data verification rule matched with the current data processing task; obtaining batch data to be processed, dividing the batch data into multiple data sets, and determining a target data set; taking each target data set as a scheduling unit, submitting a concurrent processing task; after the concurrent processing task is completed, for each target data set, verifying a processing result storage area thereof based on the data verification rule and taking an original data storage area thereof as a reference; after a file state of a target data set that fails in verification is rescanned, the target data set is taken as a scheduling unit to submit a concurrent processing task until all target data sets are successfully verified or a maximum number of retries is reached; and outputting a processing result storage area corresponding to a target data set that passes verification to obtain a batch data processing result. The scheme provided by the present application can improve data processing efficiency.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically, to a data processing method and apparatus. Background Technology

[0002] In some data processing fields, the entire data processing process is typically highly complex, encompassing multiple stages. For example, in automatic data annotation scenarios, a complete data processing chain usually includes at least several stages such as keyframe extraction, atomic subtask segmentation, bounding box detection, high-level instruction generation, and visual question-answering sample generation. These stages involve extensive video decoding, slicing, and Parquet reading and writing, as well as high-frequency calls to remote multimodal model application programming interfaces (APIs). Therefore, the overall data processing process is highly complex. Furthermore, this makes the data processing process prone to errors, and when errors occur, the entire data processing task needs to be re-executed on the entire dataset.

[0003] When dealing with massive amounts of data, and errors occur during data processing requiring a complete re-execution, the system faces the following typical problems: When an error occurs, some datasets may be processed successfully while others fail. However, known methods require re-running the entire dataset, leading to redundant processing of successfully processed data, wasting concurrent resources and remote model call quotas, resulting in low data processing efficiency. Therefore, providing a data processing method to improve efficiency has become a pressing technical problem. Summary of the Invention

[0004] This application provides a data processing method and apparatus to improve data processing efficiency. The specific technical solution is as follows.

[0005] In a first aspect, embodiments of this application provide a data processing method, the method comprising: Based on the requirements description of the data processing task, obtain the data validation rules that match the current data processing task; Acquire the batch of data to be processed, divide it into multiple datasets, and scan the file status of each dataset to determine the target dataset; Each target dataset is used as a scheduling unit to submit concurrent processing tasks; wherein, each target dataset corresponds to an original data storage area and a processing result storage area during the concurrent processing task. After the concurrent processing task is completed, for each target dataset, its processing result storage area is verified with reference to its original data storage area and based on the data verification rules. After verifying each target dataset, the target dataset that fails verification is rescanned for file status and submitted as a scheduling unit for concurrent processing tasks until all target datasets are successfully verified or the maximum number of retries is reached. Output the processing results stored in the target dataset that has been successfully verified to obtain the batch data processing results.

[0006] In one embodiment of this application, the step of scanning the file status of each dataset to determine the target dataset includes: For any dataset, compare the files in its original data storage area and the processing result storage area to determine the file status of the dataset; the file status includes: completed, partially completed, not started, or corrupted; The dataset whose file status is "not started" is identified as the target dataset.

[0007] In one embodiment of this application, the step of comparing the files in the original data storage area and the processing result storage area of ​​any dataset to determine the file status of the dataset includes: For any dataset, the initial state of the dataset is determined by checking the existence of files in its processing result storage area; the completion state of the dataset is determined by comparing the structural consistency of files in its original data storage area and processing result storage area; and the corruption state of the dataset is determined by checking the physical health of files in its processing result storage area.

[0008] In one embodiment of this application, the steps of determining the starting state of any dataset by judging the existence of files in its processing result storage area, determining the completion state of the dataset by comparing the structural consistency of files in its original data storage area and processing result storage area, and determining the corruption state of the dataset by judging the physical health of files in its processing result storage area include: For any dataset, if no processing result storage area has been generated or no valid output has been generated, the file status of the dataset is determined to be "not started". For any dataset, if the set of filenames in the preset folders in its original data storage area and processing result storage area are consistent, the file size is valid, and the column, type, and metadata verification requirements are met, the file status of the dataset is determined to be complete. For any dataset, if some processing artifacts already exist in its processing result storage area but have not yet passed the integrity check, the file status of the dataset is determined to be partially completed. For any dataset, if the processing results already exist in its processing result storage area, but there are preset file header and tail magic number errors, pattern reading failures, or abnormal file sizes, the file status of the dataset is determined to be corrupted.

[0009] In one embodiment of this application, the step of verifying the processing result storage area of ​​each target dataset by referring to its original data storage area and based on the data verification rules after the concurrent processing task is completed includes: After the concurrent processing task is completed, for each target dataset, the integrity of its processing result storage area is checked with reference to its original data storage area and based on the data verification rules.

[0010] In one embodiment of this application, the step of performing integrity verification on the processing result storage area of ​​each target dataset, with reference to its original data storage area and based on the data verification rules, includes: For each target dataset, its original data storage area is used as a reference, and the files in the processing result storage area of ​​the target dataset are subjected to structured verification based on the data verification rules. When the file structure in the processing result storage area is complete, the format is correct, and the content is valid, the integrity verification of the target dataset is determined to be passed.

[0011] In one embodiment of this application, after performing integrity verification on the processing result storage area of ​​each target dataset with reference to its original data storage area and based on the data verification rules, the method further includes: Based on the preset quality verification rules, the quality of the storage area for the processing results of the target dataset is verified. If the integrity and quality checks of the target dataset fail, the target dataset is deemed to have failed the checks.

[0012] In one embodiment of this application, after scanning the file status of each dataset to determine the target dataset, the method further includes: Perform corresponding cleanup operations on datasets whose file status is partially completed or corrupted, delete conflicting or corrupted artifacts, and correct the file status of the cleaned dataset to not started.

[0013] In one embodiment of this application, the step of submitting concurrent processing tasks using each target dataset as a scheduling unit includes: Using each target dataset as a scheduling unit, a task execution pipeline corresponding to the task type is constructed according to the task type; The task types include, but are not limited to, robot multimodal data annotation tasks, large model training data generation tasks, and visual data quality inspection tasks.

[0014] In one embodiment of this application, the step of constructing a task execution pipeline corresponding to the task type includes: The multimodal model is invoked for task processing, and robust control is implemented during the model invocation process.

[0015] Secondly, embodiments of this application provide a data processing apparatus, the apparatus comprising: The rule acquisition module is used to acquire data validation rules that match the current data processing task based on the requirement description of the data processing task. The status scanning module is used to acquire the batch data to be processed, divide it into multiple datasets, and scan the file status of each dataset to determine the target dataset; The task submission module is used to submit concurrent processing tasks with each of the target datasets as a scheduling unit; wherein, each of the target datasets corresponds to an original data storage area and a processing result storage area during the concurrent processing task. The result verification module is used to verify the processing result storage area of ​​each target dataset by referring to its original data storage area and based on the data verification rules after the concurrent processing task is completed. The task retry module is used to verify each target dataset, rescan the file status of the target dataset that failed the verification, and submit it as a scheduling unit for concurrent processing until all target datasets are successfully verified or the maximum number of retries is reached. The result output module is used to output the processing result storage area corresponding to the successfully verified target dataset to obtain the batch data processing results.

[0016] In one embodiment of this application, the state scanning module is specifically used for: For any dataset, compare the files in its original data storage area and the processing result storage area to determine the file status of the dataset; the file status includes: completed, partially completed, not started, or corrupted; The dataset whose file status is "not started" is identified as the target dataset.

[0017] In one embodiment of this application, the state scanning module is specifically used for: For any dataset, the initial state of the dataset is determined by checking the existence of files in its processing result storage area; the completion state of the dataset is determined by comparing the structural consistency of files in its original data storage area and processing result storage area; and the corruption state of the dataset is determined by checking the physical health of files in its processing result storage area.

[0018] In one embodiment of this application, the state scanning module is specifically used for: For any dataset, if no processing result storage area has been generated or no valid output has been generated, the file status of the dataset is determined to be "not started". For any dataset, if the set of filenames in the preset folders in its original data storage area and processing result storage area are consistent, the file size is valid, and the column, type, and metadata verification requirements are met, the file status of the dataset is determined to be complete. For any dataset, if some processing artifacts already exist in its processing result storage area but have not yet passed the integrity check, the file status of the dataset is determined to be partially completed. For any dataset, if the processing results already exist in its processing result storage area, but there are preset file header and tail magic number errors, pattern reading failures, or abnormal file sizes, the file status of the dataset is determined to be corrupted.

[0019] In one embodiment of this application, the result verification module is specifically used for: After the concurrent processing task is completed, for each target dataset, the integrity of its processing result storage area is checked with reference to its original data storage area and based on the data verification rules.

[0020] In one embodiment of this application, the result verification module is specifically used for: For each target dataset, its original data storage area is used as a reference, and the files in the processing result storage area of ​​the target dataset are subjected to structured verification based on the data verification rules. When the file structure in the processing result storage area is complete, the format is correct, and the content is valid, the integrity verification of the target dataset is determined to be passed.

[0021] In one embodiment of this application, the apparatus further includes: The quality verification module is used to perform quality verification on the storage area of ​​the processing results of the target dataset based on preset quality verification rules; if the integrity verification and quality verification of the target dataset fail, the verification of the target dataset is determined to be unsuccessful.

[0022] In one embodiment of this application, the apparatus further includes: The data cleaning module is used to perform corresponding cleaning operations on datasets whose file status is partially completed or corrupted, delete conflicting or corrupted artifacts, and correct the file status of the cleaned dataset to "not started".

[0023] In one embodiment of this application, the task submission module is specifically used for: Using each target dataset as a scheduling unit, a task execution pipeline corresponding to the task type is constructed according to the task type; The task types include, but are not limited to, robot multimodal data annotation tasks, large model training data generation tasks, and visual data quality inspection tasks.

[0024] In one embodiment of this application, the task submission module is specifically used for: The multimodal model is invoked for task processing, and robust control is implemented during the model invocation process.

[0025] In this embodiment, data processing can be performed on a dataset-by-dataset basis. That is, data processing can be performed through concurrent execution of multiple processes, where one dataset corresponds to one process, and each dataset corresponds to an original data storage area and a processing result storage area. This allows each dataset to be executed in isolation, ensuring that intermediate data and processing results do not affect each other, and the final processing result will not overwrite the original data. If an error occurs during the processing of any dataset, the dataset can be reprocessed based on its corresponding original data storage area and processing result storage area, without having to rerun the entire dataset. This saves concurrent resources and improves data processing efficiency. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0027] Figure 1 A schematic flowchart of a data processing method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the automatic robot data annotation process provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0029] It should be noted that the terms "comprising" and "having," and any variations thereof, in the embodiments and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0030] Known data processing methods lack a dataset-level isolation execution mechanism. When multiple concurrent tasks share an intermediate directory or output namespace, issues such as slice file conflicts, cache file overwriting, and incorrect result writes can easily occur. Furthermore, the lack of a dual-area write mechanism for both the original data area and the processed result area often results directly overwriting the original data. If the process is interrupted, a remote model returns an exception, or the file system experiences instability, it can easily lead to partially written Parquet files and partially written metadata files, making subsequent runs difficult to recover from. Additionally, known methods often retry entire batches during retries, failing to retry only the datasets that failed validation. This results in repeatedly processing successfully processed datasets, wasting concurrent resources and remote model call quotas, leading to low data processing efficiency.

[0031] Based on this, embodiments of this application disclose a data processing method and apparatus that can improve data processing efficiency. The embodiments of this application will be described in detail below.

[0032] Figure 1 This illustration shows a flowchart of a data processing method provided in an embodiment of this application. The method can be applied to electronic devices and includes the following steps: S1: Based on the requirements description of the data processing task, obtain the data verification rules that match the current data processing task.

[0033] The data processing method of this application embodiment can be applied to any scenario that requires big data processing, such as robot data annotation, large model training data generation, autonomous driving road test data processing, industrial vision data quality inspection, medical image data AI analysis, and other scenarios.

[0034] It is understandable that different application scenarios require different data processing procedures and have different data processing task requirements. In this embodiment, during data processing, data verification rules matching the current data processing task can be obtained first based on the requirement description of the data processing task.

[0035] Data validation rules are used to define criteria for judging the integrity of files, field completeness, data format, and compliance of values ​​after batch data processing. For example, global validation rules adapted to dataset units can be automatically generated based on preset business configurations (such as data structure configuration, processing flow configuration, dataset directory paradigm, industry business constraints, etc.).

[0036] S2: Obtain the batch of data to be processed, divide it into multiple datasets, and scan the file status of each dataset to determine the target dataset.

[0037] To address the issue of low efficiency in data processing, this embodiment of the application proposes processing data on a dataset-by-dataset basis. That is, data processing can be performed through multiple concurrent processes, with one process corresponding to one dataset, and each process having its own independent results directory for storing intermediate data and processing results. This allows for isolated execution of each dataset, ensuring that intermediate data and processing results do not interfere with each other, and that processing results do not overwrite the original data. If an error occurs during the processing of any dataset, the dataset can be reprocessed based on its independent results directory and the original data, without requiring a complete rerun, thus saving concurrent resources and improving data processing efficiency.

[0038] Specifically, the system can acquire batches of data to be processed and divide them into multiple datasets. For example, a configuration file can be loaded; this configuration file should at least include: the original data directory for each dataset, the number of concurrent workers (the maximum allowed number of concurrent workers), and the intermediate result directory for each dataset. It may also include: whether to overwrite the original data, the maximum number of retries, whether to stop early if the failure set remains unchanged, the failure report output path, and the target class, output column names, and metadata requirements for each task in the pipeline. The original data directory and the intermediate result directory are different, thus separating the original data area from the processing result area, ensuring that all intermediate data does not overwrite the original data during processing.

[0039] Therefore, in this embodiment, the batch data to be processed can be obtained based on the dataset path information and divided into multiple datasets. For example, the obtained batch data can be divided into multiple datasets of the same size according to the preset size of each dataset.

[0040] It's understandable that the file states of multiple datasets are not entirely the same. For example, some datasets may have already started processing while others haven't; or some datasets may have been completed while others haven't. Therefore, to improve data processing efficiency, the processing methods differ depending on the file state of the dataset.

[0041] In this embodiment, after dividing the batch data into multiple datasets, the file status of each dataset can be scanned to determine the target dataset. The target dataset is the dataset that can be directly processed further and can be referred to as the unstarted dataset.

[0042] In one implementation, for any dataset, the files in its original data storage area and the processing result storage area can be compared to determine the file status of the dataset; the file status includes: completed, partially completed, not started, or corrupted; then the dataset with the file status of not started is determined as the target dataset.

[0043] For example, for any dataset, the initial state of the dataset can be determined by checking the existence of files in its processing result storage area, the completion state of the dataset can be determined by comparing the structural consistency of files in its original data storage area and processing result storage area, and the corruption state of the dataset can be determined by checking the physical health of files in its processing result storage area.

[0044] In other words, if no file exists in the processing result storage area corresponding to any dataset, it indicates that the dataset has not yet started processing; if a file exists in the processing result storage area corresponding to any dataset, it indicates that the dataset has started processing. If the file structure consistency between the processing result storage area and the original data storage area meets the requirements, it indicates that the dataset has been processed; otherwise, processing is not yet complete. If the physical health of the files in the processing result storage area of ​​any dataset meets the requirements, it indicates that the dataset is not corrupted; otherwise, it indicates that the dataset is corrupted.

[0045] Specifically, the file status of a dataset can be identified according to the following steps: For any dataset, if no processing result storage area has been generated or no valid output has been generated, the file status of the dataset is determined to be "not started"; For any dataset, if the filename set of the preset folders in its original data storage area and processing result storage area is consistent, the file size is valid, and the column, type, and metadata verification requirements are met, the file status of the dataset is determined to be "completed"; For any dataset, if some processing products exist in its processing result storage area but have not yet passed the integrity verification, the file status of the dataset is determined to be "partially completed"; For any dataset, if processing results exist in its processing result storage area but there are preset file header / tail magic number errors, mode reading failures, or abnormal file sizes, the file status of the dataset is determined to be "corrupted".

[0046] For completed datasets, the system can skip them directly; for partially completed or corrupted datasets, the system can clean up conflicts or corrupted artifacts before execution, thus enabling an automatic identification and cleanup mechanism for corrupted outputs; for datasets that have not yet started, they are used as target datasets and directly entered into concurrent processing.

[0047] In one embodiment of this application, a corresponding cleaning operation can be performed on a dataset whose file status is partially completed or damaged, deleting conflicting or damaged artifacts, and correcting the file status of the cleaned dataset to "not started," after which the corresponding processing operation can be performed on the dataset.

[0048] S3: Submit concurrent processing tasks using each target dataset as a scheduling unit; each target dataset corresponds to an original data storage area and a processing result storage area during the concurrent processing task.

[0049] In this embodiment, concurrent tasks can be submitted on a dataset-by-dataset basis. That is, each dataset serves as a scheduling unit, managed by an independent process, such as a process pool. Furthermore, each target dataset corresponds to an original data storage area and a processing result storage area during concurrent processing, thereby preventing the original data from being corrupted and avoiding mutual contamination of the runtime environment.

[0050] The aforementioned local process pool can be replaced by a thread pool, a distributed task queue, a container orchestration platform, or a cluster scheduling platform, as long as the core idea of ​​"dataset-level isolated execution + failure set retry" is still maintained.

[0051] S4: After the concurrent processing task is completed, for each target dataset, the processing result storage area is verified with reference to its original data storage area and based on the data verification rules.

[0052] In this embodiment of the application, after the concurrent processing task is completed, for each target dataset, its original data storage area is used as a reference and its processing result storage area is verified based on data verification rules to verify whether the data processing result meets the requirements.

[0053] In one implementation, after concurrent processing tasks are completed, for each target dataset, integrity verification can be performed on its processing result storage area using its original data storage area as a reference and based on data verification rules. In other words, dataset verification can be performed at the structural level, such as verifying its file structure, data schema, and physical integrity.

[0054] Specifically, for each target dataset, its original data storage area can be used as a reference, and the files in the processing result storage area of ​​the target dataset can be structurally verified based on data verification rules. When the file structure in the processing result storage area is complete, the format is correct, and the content is valid, the integrity verification of the target dataset is determined to be passed.

[0055] It is understandable that, in addition to ensuring structural integrity, the quality of the dataset processing results is also related to its specific content. Therefore, in this embodiment, quality verification can also be performed on the dataset. Specifically, for each target dataset, its original data storage area can be used as a reference, and its processing result storage area can be verified for integrity based on data verification rules. Then, based on preset quality verification rules, the processing result storage area of ​​the target dataset can be verified for quality. If both the integrity verification and quality verification of the target dataset fail, the target dataset is determined to have failed verification. In other words, only when both the integrity verification and quality verification pass is the target dataset considered to have successfully verified.

[0056] Quality verification can be used to check data content, business logic, and data accuracy. By performing integrity and quality verification on the dataset simultaneously, the accuracy of the final data processing results can be improved.

[0057] S5: After verifying each target dataset, the target datasets that fail verification are rescanned for file status and submitted as a scheduling unit for concurrent processing tasks until all target datasets are successfully verified or the maximum number of retries is reached.

[0058] In this embodiment, when the verification finds that some datasets have failed, instead of re-executing all datasets, the target datasets that failed verification can be re-scanned for file status and submitted as a scheduling unit for concurrent processing until all target datasets have been successfully verified or the maximum number of retries has been reached. Specifically, the following operations can be performed: 1. Form a "failure set"; 2. Only resubmit the failed set to the concurrent executor; 3. Verify the processing results again; 4. If the failure set in this round is exactly the same as the failure set in the previous round, the system is considered to have made no progress and will be stopped prematurely; 5. Write the final failed dataset and the reason for the failure into the failure report.

[0059] This "failure set shrinking retry" mechanism can significantly reduce the number of times remote models are repeatedly called compared to a full retry, making it particularly suitable for scenarios with long videos, large datasets, and high API costs.

[0060] S6: Output the processing result storage area corresponding to the successfully verified target dataset to obtain the batch data processing result.

[0061] In this embodiment, once the entire dataset has been processed, the processing result storage area corresponding to the successfully verified target dataset can be output to obtain the batch data processing results. In other words, the disk persistence can be securely upgraded, transforming the processing result storage area into a formal data area. Furthermore, the completion marker file can be replaced with a database status table, message queue confirmation messages, object storage metadata tags, or transaction logs.

[0062] Therefore, the official data always comes from the processed results area that has passed verification, while the original backup is retained to support subsequent rollback.

[0063] In this embodiment, data processing can be performed on a dataset-by-dataset basis. That is, data processing can be performed through concurrent execution of multiple processes, where one dataset corresponds to one process, and each dataset corresponds to an original data storage area and a processing result storage area. This allows each dataset to be executed in isolation, ensuring that intermediate data and processing results do not affect each other, and the final processing result will not overwrite the original data. If an error occurs during the processing of any dataset, the dataset can be reprocessed based on its corresponding original data storage area and processing result storage area, without having to rerun the entire dataset. This saves concurrent resources and improves data processing efficiency.

[0064] Specifically, this application has the following advantages: 1. Bind "concurrent execution" with "dataset directory semantics".

[0065] This application does not schedule tasks in an abstract way, but designs a special scheduling and disk-based mechanism for the inherent `data / videos / meta` structure of large datasets.

[0066] 2. Protect original data in high-concurrency scenarios.

[0067] All new results are first written to the processing result area and the metadata temporary storage area to avoid directly overwriting the original data and causing irreversible damage.

[0068] 3. The verification logic is automatically consistent with the pipeline configuration.

[0069] By automatically inferring the required columns and metadata files from the pipeline, the complexity of manually maintaining multiple validation scripts is reduced.

[0070] 4. Only retry the set of failed attempts.

[0071] It performs zero-duplication processing on successful datasets and accurately retryes failed datasets, significantly reducing overall time and model call costs.

[0072] 5. It has the ability to automatically identify and clean damaged workpieces.

[0073] It can proactively remove invalid Parquet files before retrying, preventing erroneous artifacts from interfering with subsequent operations for an extended period.

[0074] 6. It has the capability for security upgrades and rollbacks.

[0075] The processing results are only upgraded to official data after passing verification, while an "old" backup is retained, enhancing the controllability of the production environment.

[0076] 7. Applicable to multiple types of task chains.

[0077] It can be used for keyframe extraction, atomic task segmentation, high-level instruction generation, and also for generating visual question-answering samples from multiple datasets, making it versatile.

[0078] 8. More suitable for large-scale remote model invocation scenarios.

[0079] By employing caching, structured response validation, exponential backoff retries, and failure reporting mechanisms, the system can maintain stable operation even when processing hundreds or thousands of datasets concurrently.

[0080] In one embodiment of this application, the step of submitting concurrent processing tasks using each target dataset as a scheduling unit may include: using each target dataset as a scheduling unit, constructing a task execution pipeline corresponding to the task type according to the task type; wherein, the task type includes, but is not limited to, robot multimodal data annotation tasks, large model training data generation tasks, and visual data quality inspection tasks. That is, the data processing method provided in this application embodiment can be applied to various data processing scenarios, such as robot multimodal data annotation tasks, large model training data generation tasks, and visual data quality inspection tasks.

[0081] Specifically, the steps mentioned above for constructing a task execution pipeline corresponding to a task type may include: calling a multimodal model for task processing and performing robust control during the model call process.

[0082] In other words, in a concurrent task execution pipeline, after data preprocessing (such as video slicing, frame sampling, etc.) is completed, the robust model inference step can begin. This step can specifically include the following steps: Available model service endpoints are selected by polling the address pool. Initiate an inference request with an exponential backoff retry mechanism; Perform structured validation on the returned JSON response; The valid results are passed to the subsequent parsing and writing steps.

[0083] The robust control built into this step allows upper-layer tasks to stably obtain model output without being aware of network jitter, service rate limiting, or node failures.

[0084] The following example, using a robot data annotation scenario, illustrates the data processing method provided in this application. Figure 2 As shown, the process includes the following steps: S101: Loading processing configuration; Specifically, the system can read the YAML configuration file to obtain the following parameters: A list of dataset paths `lerobot_paths` or multiple dataset directories; The number of concurrent workers is `num_workers`. Intermediate results directory `intermediate_results_dir`; Should the original data be overwritten? Maximum number of retry rounds `max_retry_rounds`; Should `stop_retry_when_no_progress` be stopped early if the failure set remains unchanged? The failure report output path is `failed_report_path`; The target class, output column names, and metadata requirements for each task in the pipeline.

[0085] In one implementation, configuration can support high concurrency parameters. For example, `num_workers: 500` can be set in `gemini_subtask / configs / high_level_instruction_100h.yaml`.

[0086] Alternatively, set `num_workers: 1000` in `gemini_subtask / configs / high_level_instruction_8595h.yaml`.

[0087] The above configuration indicates that this application is not intended for serial execution of a single dataset, but rather for batch processing of ultra-large-scale datasets.

[0088] S102: Automatically infer verification rules based on the pipeline; This application does not require users to manually write a separate validator for each pipeline; instead, it automatically infers the validator from the pipeline. Required output columns, such as `atomic_task_index`, `high_level_instruction_index`, and `is_key_frame`; Required integer columns, for example, `atomic_task_index` must be `int64` and cannot contain null values; Required metadata files, such as `meta / tasks_new.jsonl` and `meta / info_new.json`.

[0089] This step ensures that the "processing logic" and "verification logic" are driven by the same configuration source, thereby avoiding misjudgments caused by human error in high-concurrency scenarios.

[0090] S103: Dataset Status Scanning and Classification; Before formally submitting concurrent tasks, the system can perform state identification on each dataset. Specifically, the datasets can be categorized into at least the following types: 1. Completed dataset: The filename set `data_new / chunk-000` is consistent with `data / chunk-000`, the file size is valid, and the column, type, and metadata validation requirements are met.

[0091] 2. Partially completed dataset: Some processed products already exist, but the integrity requirements have not yet been met.

[0092] 3. Dataset not started: The processing result area has not yet been generated or no valid output has been generated.

[0093] 4. Corrupted dataset: The processing results already exist, but some files (such as Parquet files) have incorrect magic numbers at the beginning and end, mode reading failures, or abnormal file sizes.

[0094] For completed datasets, the system skips them directly; for partially completed or corrupted datasets, the system cleans up conflicts or corrupted artifacts before execution; for datasets that have not yet started, the system proceeds directly to concurrent processing.

[0095] S104: Submit concurrent tasks in units of datasets; This application uses the "dataset root directory" as the smallest scheduling unit, rather than simply using a single frame, single segment, or single file. Each dataset is handled by an independent process, preferably using a process pool. Its advantages are: The task object can be re-instantiated within the child process, avoiding mutual pollution of the runtime environment; This allows each dataset to have its own independent temporary working directory; It can aggregate tokens, time spent, exceptions, and completion status by dataset dimension; Retry, cleanup, and failure reporting can be managed centrally within the parent process.

[0096] S105: Perform multi-stage annotation within a separate working directory; Each concurrent worker process creates a separate working directory for a single dataset, for example: `<intermediate_results_dir> / <dataset_name> / episode_xxxxxx / <task_name> / ` Within this separate working directory, tasks can perform the following operations: Slice the video and generate `clips_metadata.json`; Cache the model response JSON file for a single fragment; Generate a cache for high-level instructions. <model>.json; Record token usage logs; Generate debugging visualization files.

[0097] Because each dataset has its own independent working directory, multiple concurrent worker processes will not compete for the same path, nor will they overwrite each other due to temporary files having the same name.

[0098] S106: Dual-zone temporary storage write; This application adopts a method that separates the "raw data area" and the "processing result area". Specifically: Raw data area: `data / chunk-000`, `meta / tasks.jsonl`, `meta / info.json`; Processing results area: `data_new / chunk-000`, `meta / tasks_new.jsonl`, `meta / info_new.json`.

[0099] During processing, all newly added columns and new task mappings are first written to the processing result area, without directly overwriting the original data area. Preferably: 1. Parquet output is written to `data_new / chunk-000 / episode_xxxxxx.parquet`; 2. Task mappings are written to `meta / tasks_new.jsonl`; 3. Write the information description to `meta / info_new.json`; 4. The metadata file uses an atomic replacement method of "temporary file writing + `flush` + `fsync` + `os.replace`".

[0100] The purpose of the above mechanism is to ensure that even if the process crashes during high-concurrency execution, the original data area will not be corrupted, and it can prevent the half-written metadata file from existing for a long time.

[0101] S107: Robust control of model invocation; In one implementation, this application also employs the following high-concurrency adaptation measures for remote model calls: 1. Perform structured validation on the JSON response returned by the model; if the response format is incorrect, automatically retry. 2. Implement exponential backoff and retries for network errors, request errors, and server errors; 3. Perform a cache hit check on the result file; if the cache is valid, reuse it directly to avoid repeated calls. 4. The access address of the model can be polled from a pool or randomly distributed to reduce the pressure on a single server endpoint; 5. Log the token consumption to facilitate the calculation of resource costs in high-concurrency environments.

[0102] This step couples "high-concurrency scheduling" and "remote multimodal model invocation" into a unified fault-tolerant link, rather than simply relying on process pools.

[0103] S108: Verification of the integrity of processing results; After the parent process completes a round of concurrent processing, it can perform integrity checks on all datasets or the entire target dataset. The checks should include at least the following: 1. Are the file sets `data / chunk-000` and `data_new / chunk-000` consistent? 2. Is the file size in `data_new / chunk-000` positive? 3. Check if all required output columns exist; 4. Check if the required integer column is empty and if it is strictly `int64`; 5. The required metadata file exists and is not empty; 6. Preferably, the Parquet file is additionally checked for the start and end magic numbers and whether the pattern is read correctly.

[0104] If a corrupted Parquet file is found in the processing results area and a file with the same name exists in the original data area, the system will delete the corrupted processing results file so that only that part will be regenerated in the next round.

[0105] S109: Retry with shrinking failure set; When the validation fails for some datasets, this application does not re-execute on all datasets, but instead performs the following operations: 1. Form a "failure set"; 2. Only resubmit the failed set to the concurrent executor; 3. Verify the processing results again; 4. If the failure set in this round is exactly the same as the failure set in the previous round, the system is considered to have made no progress and will be stopped prematurely; 5. Write the final failed dataset and the reason for the failure into the failure report.

[0106] This "failure set shrinking retry" mechanism can significantly reduce the number of times remote models are repeatedly called compared to a full retry, making it particularly suitable for scenarios with long videos, large datasets, and high API costs.

[0107] S110: Safety upgrade for disk placement; Only after the entire dataset has passed validation will this application upgrade the processing results area to the official data area. Specifically, the following upgrade strategy can be implemented: 1. Rename `data / ` to `data_old / `; 2. Rename `data_new / ` to `data / `; 3. If `tasks_new.jsonl` exists, back up `tasks.jsonl` to `tasks_old.jsonl`, and then upgrade `tasks_new.jsonl` to `tasks.jsonl`; 4. If `info_new.json` exists, back up `info.json` to `info_old.json`, and then upgrade `info_new.json` to `info.json`.

[0108] Therefore, the official data always comes from the processed results area that has passed verification, while the original backup is retained to support subsequent rollback.

[0109] In one embodiment of this application, the data processing method provided in this application can also be used in a visual question-answering data generation scenario. In this case, each dataset can first be output as an independent result fragment, and then merged uniformly, as follows: 1. Each dataset is output by an independent worker process in `_shards / `<safe_dataset_name> .json or multiple split files; 2. Once a dataset has been processed, write the completion marker to the corresponding file, such as `_status / `.<safe_dataset_name> .done.json; 3. When resuming from a breakpoint, the system prioritizes identifying completed markers and result slices, and skips completed datasets directly; 4. For datasets with residual fragments but no completion markers, the system automatically cleans up the residual artifacts and reruns the dataset; 5. After all datasets have been processed, the results of the successful datasets are split and merged into the final output.

[0110] This implementation method avoids multiple concurrent worker processes writing to the same final result file at the same time, thereby reducing the risk of write lock contention and result overwriting.

[0111] Figure 3 This illustration shows a schematic diagram of a data processing apparatus according to an embodiment of this application. The apparatus includes: The rule acquisition module 310 is used to acquire data verification rules that match the current data processing task based on the requirement description of the data processing task. The status scanning module 320 is used to acquire the batch data to be processed, divide it into multiple datasets, and scan the file status of each dataset to determine the target dataset; The task submission module 330 is used to submit concurrent processing tasks with each of the target datasets as a scheduling unit; wherein, each of the target datasets corresponds to an original data storage area and a processing result storage area during the concurrent processing task. The result verification module 340 is used to verify the processing result storage area of ​​each target dataset by referring to its original data storage area and based on the data verification rules after the concurrent processing task is completed. The task retry module 350 is used to verify each of the target datasets, rescan the file status of the target datasets that fail the verification, and submit them as a scheduling unit for concurrent processing until all target datasets are successfully verified or the maximum number of retries is reached. The result output module 360 ​​is used to output the processing result storage area corresponding to the successfully verified target dataset to obtain the batch data processing results.

[0112] In one embodiment of this application, the state scanning module 320 is specifically used for: For any dataset, compare the files in its original data storage area and the processing result storage area to determine the file status of the dataset; the file status includes: completed, partially completed, not started, or corrupted; The dataset whose file status is "not started" is identified as the target dataset.

[0113] In one embodiment of this application, the state scanning module 320 is specifically used for: For any dataset, the initial state of the dataset is determined by checking the existence of files in its processing result storage area; the completion state of the dataset is determined by comparing the structural consistency of files in its original data storage area and processing result storage area; and the corruption state of the dataset is determined by checking the physical health of files in its processing result storage area.

[0114] In one embodiment of this application, the state scanning module 320 is specifically used for: For any dataset, if no processing result storage area has been generated or no valid output has been generated, the file status of the dataset is determined to be "not started". For any dataset, if the set of filenames in the preset folders in its original data storage area and processing result storage area are consistent, the file size is valid, and the column, type, and metadata verification requirements are met, the file status of the dataset is determined to be complete. For any dataset, if some processing artifacts already exist in its processing result storage area but have not yet passed the integrity check, the file status of the dataset is determined to be partially completed. For any dataset, if the processing results already exist in its processing result storage area, but there are preset file header and tail magic number errors, pattern reading failures, or abnormal file sizes, the file status of the dataset is determined to be corrupted.

[0115] In one embodiment of this application, the result verification module 340 is specifically used for: After the concurrent processing task is completed, for each target dataset, the integrity of its processing result storage area is checked with reference to its original data storage area and based on the data verification rules.

[0116] In one embodiment of this application, the result verification module 340 is specifically used for: For each target dataset, its original data storage area is used as a reference, and the files in the processing result storage area of ​​the target dataset are subjected to structured verification based on the data verification rules. When the file structure in the processing result storage area is complete, the format is correct, and the content is valid, the integrity verification of the target dataset is determined to be passed.

[0117] In one embodiment of this application, the apparatus further includes: The quality verification module is used to perform quality verification on the storage area of ​​the processing results of the target dataset based on preset quality verification rules; if the integrity verification and quality verification of the target dataset fail, the verification of the target dataset is determined to be unsuccessful.

[0118] In one embodiment of this application, the apparatus further includes: The data cleaning module is used to perform corresponding cleaning operations on datasets whose file status is partially completed or corrupted, delete conflicting or corrupted artifacts, and correct the file status of the cleaned dataset to "not started".

[0119] In one embodiment of this application, the task submission module 330 is specifically used for: Using each target dataset as a scheduling unit, a task execution pipeline corresponding to the task type is constructed according to the task type; The task types include, but are not limited to, robot multimodal data annotation tasks, large model training data generation tasks, and visual data quality inspection tasks.

[0120] In one embodiment of this application, the task submission module 330 is specifically used for: The multimodal model is invoked for task processing, and robust control is implemented during the model invocation process.

[0121] In this embodiment, data processing can be performed on a dataset-by-dataset basis. That is, data processing can be performed through concurrent execution of multiple processes, where one dataset corresponds to one process, and each dataset corresponds to an original data storage area and a processing result storage area. This allows each dataset to be executed in isolation, ensuring that intermediate data and processing results do not affect each other, and the final processing result will not overwrite the original data. If an error occurs during the processing of any dataset, the dataset can be reprocessed based on its corresponding original data storage area and processing result storage area, without having to rerun the entire dataset. This saves concurrent resources and improves data processing efficiency.

[0122] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing this application.

[0123] Those skilled in the art will understand that the modules in the apparatus of the embodiments can be distributed in the apparatus of the embodiments as described in the embodiments, or they can be located in one or more devices different from this embodiment with corresponding changes. The modules of the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.< / model>

Claims

1. A data processing method, characterized in that, The method includes: Based on the requirements description of the data processing task, obtain the data validation rules that match the current data processing task; Acquire the batch of data to be processed, divide it into multiple datasets, and scan the file status of each dataset to determine the target dataset; Each target dataset is used as a scheduling unit to submit concurrent processing tasks; wherein, each target dataset corresponds to an original data storage area and a processing result storage area during the concurrent processing task. After the concurrent processing task is completed, for each target dataset, its processing result storage area is verified with reference to its original data storage area and based on the data verification rules. After verifying each target dataset, the target dataset that fails verification is rescanned for file status and submitted as a scheduling unit for concurrent processing tasks until all target datasets are successfully verified or the maximum number of retries is reached. Output the processing results stored in the target dataset that has been successfully verified to obtain the batch data processing results; The step of scanning the file status of each dataset to determine the target dataset includes: For any dataset, compare the files in its original data storage area and the processing result storage area to determine the file status of the dataset; the file status includes: completed, partially completed, not started, or corrupted; The dataset whose file status is "not started" is identified as the target dataset.

2. The method according to claim 1, characterized in that, The step of comparing the files in the original data storage area and the processing result storage area of ​​any dataset to determine the file status of the dataset includes: For any dataset, the initial state of the dataset is determined by checking the existence of files in its processing result storage area; the completion state of the dataset is determined by comparing the structural consistency of files in its original data storage area and processing result storage area; and the corruption state of the dataset is determined by checking the physical health of files in its processing result storage area.

3. The method according to claim 2, characterized in that, The steps for determining the initial state of a dataset by checking the existence of files in its processing result storage area, determining the completion state of the dataset by comparing the structural consistency of files in its original data storage area and processing result storage area, and determining the corruption state of the dataset by checking the physical health of files in its processing result storage area include: For any dataset, if no processing result storage area has been generated or no valid output has been generated, the file status of the dataset is determined to be "not started". For any dataset, if the set of filenames in the preset folders in its original data storage area and processing result storage area are consistent, the file size is valid, and the column, type, and metadata verification requirements are met, the file status of the dataset is determined to be complete. For any dataset, if some processing artifacts already exist in its processing result storage area but have not yet passed the integrity check, the file status of the dataset is determined to be partially completed. For any dataset, if the processing results already exist in its processing result storage area, but there are preset file header and tail magic number errors, pattern reading failures, or abnormal file sizes, the file status of the dataset is determined to be corrupted.

4. The method according to claim 1, characterized in that, The step of verifying the processing result storage area of ​​each target dataset by referring to its original data storage area and based on the data verification rules after the concurrent processing task is completed includes: After the concurrent processing task is completed, for each target dataset, the integrity of its processing result storage area is checked with reference to its original data storage area and based on the data verification rules.

5. The method according to claim 4, characterized in that, The step of performing integrity verification on the processing result storage area of ​​each target dataset, with reference to its original data storage area and based on the data verification rules, includes: For each target dataset, its original data storage area is used as a reference, and the files in the processing result storage area of ​​the target dataset are subjected to structured verification based on the data verification rules. When the file structure in the processing result storage area is complete, the format is correct, and the content is valid, the integrity verification of the target dataset is determined to be passed.

6. The method according to claim 4, characterized in that, After performing integrity verification on the processing result storage area of ​​each target dataset, using its original data storage area as a reference and based on the data verification rules, the method further includes: Based on the preset quality verification rules, the quality of the storage area for the processing results of the target dataset is verified. If the integrity and quality checks of the target dataset fail, the target dataset is deemed to have failed the checks.

7. The method according to any one of claims 1-4, characterized in that, After scanning the file status of each dataset to determine the target dataset, the method further includes: Perform corresponding cleanup operations on datasets whose file status is partially completed or corrupted, delete conflicting or corrupted artifacts, and correct the file status of the cleaned dataset to not started.

8. The method according to claim 1, characterized in that, The step of submitting concurrent processing tasks using each of the target datasets as a scheduling unit includes: Each target dataset is used as a scheduling unit, and a task execution pipeline corresponding to the task type is constructed according to the task type; the task types include, but are not limited to, robot multimodal data annotation tasks, large model training data generation tasks, and visual data quality inspection tasks.

9. The method according to claim 8, characterized in that, The step of constructing the task execution pipeline corresponding to the task type includes: The multimodal model is invoked for task processing, and robust control is implemented during the model invocation process.

10. A data processing apparatus, characterized in that, The device includes: The rule acquisition module is used to acquire data validation rules that match the current data processing task based on the requirement description of the data processing task. The status scanning module is used to acquire the batch data to be processed, divide it into multiple datasets, and scan the file status of each dataset to determine the target dataset; The task submission module is used to submit concurrent processing tasks with each of the target datasets as a scheduling unit; wherein, each of the target datasets corresponds to an original data storage area and a processing result storage area during the concurrent processing task. The result verification module is used to verify the processing result storage area of ​​each target dataset by referring to its original data storage area and based on the data verification rules after the concurrent processing task is completed. The task retry module is used to verify each target dataset, rescan the file status of the target dataset that failed the verification, and submit it as a scheduling unit for concurrent processing until all target datasets are successfully verified or the maximum number of retries is reached. The result output module is used to output the processing result storage area corresponding to the target dataset that has been successfully verified, and obtain the batch data processing results; The status scanning module is specifically used to: for any dataset, compare the files in its original data storage area and the processing result storage area to determine the file status of the dataset; the file status includes: completed, partially completed, not started, or corrupted; and determine the dataset with the file status of not started as the target dataset.

11. The apparatus according to claim 10, characterized in that, The state scanning module is specifically used for: For any dataset, the initial state of the dataset is determined by checking the existence of files in its processing result storage area; the completion state of the dataset is determined by comparing the structural consistency of files in its original data storage area and processing result storage area; and the corruption state of the dataset is determined by checking the physical health of files in its processing result storage area.

12. The apparatus according to claim 11, characterized in that, The state scanning module is specifically used for: For any dataset, if no processing result storage area has been generated or no valid output has been generated, the file status of the dataset is determined to be "not started". For any dataset, if the set of filenames in the preset folders in its original data storage area and processing result storage area are consistent, the file size is valid, and the column, type, and metadata verification requirements are met, the file status of the dataset is determined to be complete. For any dataset, if some processing artifacts already exist in its processing result storage area but have not yet passed the integrity check, the file status of the dataset is determined to be partially completed. For any dataset, if the processing results already exist in its processing result storage area, but there are preset file header and tail magic number errors, pattern reading failures, or abnormal file sizes, the file status of the dataset is determined to be corrupted.

13. The apparatus according to claim 10, characterized in that, The result verification module is specifically used for: After the concurrent processing task is completed, for each target dataset, the integrity of its processing result storage area is checked with reference to its original data storage area and based on the data verification rules.

14. The apparatus according to claim 13, characterized in that, The result verification module is specifically used for: For each target dataset, its original data storage area is used as a reference, and the files in the processing result storage area of ​​the target dataset are subjected to structured verification based on the data verification rules. When the file structure in the processing result storage area is complete, the format is correct, and the content is valid, the integrity verification of the target dataset is determined to be passed.

15. The apparatus according to claim 13, characterized in that, The device further includes a quality verification module, which performs quality verification on the processing result storage area of ​​the target dataset based on preset quality verification rules; when the integrity verification and quality verification of the target dataset fail, the verification of the target dataset is determined to be unsuccessful.

16. The apparatus according to any one of claims 11-13, characterized in that, The device further includes a data cleaning module, used to perform corresponding cleaning operations on datasets whose file status is partially completed or damaged, delete conflicting or damaged products, and correct the file status of the cleaned dataset to not started.

17. The apparatus according to claim 10, characterized in that, The task submission module is specifically used for: Using each target dataset as a scheduling unit, a task execution pipeline corresponding to the task type is constructed according to the task type; The task types include, but are not limited to, robot multimodal data annotation tasks, large model training data generation tasks, and visual data quality inspection tasks.

18. The apparatus according to claim 17, characterized in that, The task submission module is specifically used for: The multimodal model is invoked for task processing, and robust control is implemented during the model invocation process.

Images