A data processing method and system based on the separation of elastic scheduling and pre-labeling

Abstract

This invention discloses a data processing method based on the separation of elastic scheduling and pre-labeling, comprising: acquiring data processing logic, encapsulating the data processing logic into a computing unit image and storing it; acquiring computing unit registration information and generating computing units according to the computing unit registration information; acquiring orchestration instructions and generating a data processing stream containing pre-labeled computing units according to the orchestration instructions; responding to a trigger instruction and pulling the computing unit images corresponding to each computing unit from the image storage system according to the execution order and dependencies; dynamically acquiring runtime resources from the resource pool according to runtime resource requirements to execute instances of computing unit images; wherein, the pre-labeled computing unit performs automatic inference processing on the input raw data in its runtime resources, outputs pre-labeled data and stores it in the data storage system; acquiring pre-labeled data from the data storage system and loading it into the interactive interface, acquiring operation data based on the pre-labeled data input, and generating target labeling result data.

Description

Technical Field

[0001] This invention relates to the fields of intelligent driving and cloud computing technology, specifically to a data processing method and system based on the separation of elastic scheduling and pre-labeling. Background Technology

[0002] Intelligent driving data annotation technology aims to provide high-quality labeled samples for perception model training, playing a fundamental supporting role in autonomous driving perception, behavior prediction, and decision-making planning. Currently, with the continuous expansion of the perception dimensions of intelligent driving systems, data annotation tasks have evolved from traditional two-dimensional bounding box annotation to more complex forms such as three-dimensional point cloud annotation, time-series annotation, and multi-sensor fusion annotation, placing higher demands on annotation efficiency and data consistency.

[0003] In the prior art, for example, Chinese invention patent application CN115408356A discloses an autonomous driving data processing system, method, device, and storage medium. This system stores sensor data through a storage system on a cloud server and utilizes multiple task processing tools deployed on a distributed server to acquire and process target sensor data in parallel, thereby improving the processing efficiency of large-scale autonomous driving data. However, this method mainly focuses on the storage and parallel processing architecture of sensor data and does not involve automated reasoning and interactive refinement mechanisms for annotation tasks. After data processing is completed, manual annotation operations are still required one by one. In practical applications where the data scale is growing rapidly, annotation efficiency is still limited by the speed of manual operation and cannot meet the timeliness requirements of model iteration for labeled samples.

[0004] In addition, existing rule-based or semi-automated annotation assistance methods generally suffer from insufficient generalization ability. That is, for different sensor types, different scene categories or different annotation granularities, preprocessing logic and interaction processes need to be configured separately, and there is a lack of a unified annotation task orchestration and resource scheduling mechanism. Summary of the Invention

[0005] In order to overcome the technical defects of existing technologies, such as the poor data flow links in intelligent driving and the difficulty in elastically scaling up and down according to data volume fluctuations, this invention provides a data processing method and system based on the separation of elastic scheduling and pre-labeling.

[0006] To solve the above problems, the present invention is implemented according to the following technical solution:

[0007] In a first aspect, the present invention provides a data processing method based on the separation of elastic scheduling and pre-labeling, comprising:

[0008] Obtain the data processing logic, encapsulate the data processing logic into a computing unit image, and store it;

[0009] Obtain computing unit registration information, and generate computing units based on the computing unit registration information; the computing unit registration information includes the storage address and runtime resource requirements of the computing unit image;

[0010] Obtain orchestration instructions, and generate a data processing flow containing pre-annotated computational units based on the orchestration instructions; the data processing flow defines the execution order and dependencies of each computational unit;

[0011] In response to the trigger command, the computing unit images corresponding to each computing unit are retrieved from the image storage system according to the execution order and dependencies.

[0012] Based on the aforementioned runtime resource requirements, runtime resources are dynamically acquired from the resource pool to execute instances of the computing unit image; wherein, the pre-labeled computing unit performs automatic inference processing on the input raw data within its runtime resources, outputs pre-labeled data, and stores it in the data storage system;

[0013] The pre-labeled data is obtained from the data storage system and loaded into the interactive interface. Operation data based on the pre-labeled data is obtained and target labeling result data is generated.

[0014] In conjunction with the first aspect, the present invention provides a first specific implementation of the first aspect, specifically, the step of obtaining computing unit registration information and generating computing units based on the computing unit registration information includes:

[0015] Obtain the image repository address, version number, and resource configuration parameters of the computing unit; the resource configuration parameters include the number of CPU cores, memory capacity, whether a GPU is required, and the size of the GPU memory.

[0016] Create a computing unit on the data management platform that corresponds to the image repository address and version number, and associate it with the resource configuration parameters.

[0017] In conjunction with the first aspect, the present invention provides a second specific implementation of the first aspect. Specifically, the step of dynamically acquiring runtime resources from the resource pool according to the runtime resource requirements to execute an instance of the computing unit image includes:

[0018] Obtain the resource configuration parameters of each computing unit, including the number of CPU cores, memory capacity, and GPU memory size;

[0019] The resource configuration parameters are converted into container resource quotas. Based on the container resource quotas, runtime resources are dynamically requested in the computing cluster, and the corresponding container instances are started.

[0020] Here, is the container instance started by the pre-labeling calculation unit, which loads the pre-labeling model, performs model inference on the input raw data, and outputs pre-labeled data;

[0021] The pre-labeled data is stored independently in object storage in a structured data format, separated from the original data, and associated with it through timestamps;

[0022] Monitor the execution status of each container instance and automatically release the runtime resources it occupies after the task is completed.

[0023] In conjunction with the first aspect, the present invention provides a third specific implementation of the first aspect. Specifically, the step of obtaining the pre-annotated data from the data storage system and loading it into the interactive interface, obtaining operation data input based on the pre-annotated data, and generating target annotation result data includes:

[0024] In response to the annotation task start command, the corresponding pre-annotation data is loaded from the object storage according to the timestamp of the current annotation frame; the pre-annotation data includes the position information, size information, orientation angle, category label and confidence level of the target detection box;

[0025] The pre-annotated data is displayed in the interactive interface of the annotation platform;

[0026] Obtain correction and supplementary operation data input by the annotation personnel based on the pre-annotated data;

[0027] The pre-labeled data is updated based on the correction and supplementation operation data to generate the target labeling result data.

[0028] In conjunction with the first aspect, the present invention provides a fourth embodiment of the first aspect, which specifically further includes:

[0029] Obtain historical resource usage data for each computing unit; the historical resource usage data includes actual CPU usage, actual memory usage, and actual GPU memory usage;

[0030] The historical resource usage data is analyzed to generate resource recommendation values;

[0031] The resource configuration parameters of the corresponding computing unit are updated based on the resource recommendation value, and the updated resource configuration parameters take effect when the next data processing stream is triggered.

[0032] In conjunction with the first aspect, the present invention provides a fifth embodiment of the first aspect, which specifically further includes:

[0033] Obtain the estimated execution time of each task corresponding to each computing unit;

[0034] The estimated execution time is compared with a preset threshold, and the number of parallel instances of the corresponding computing unit is automatically adjusted according to the comparison result; wherein, if the estimated execution time is less than the first threshold, a first parallel quantity is configured; if the estimated execution time is greater than the second threshold, a second parallel quantity is configured, and the second parallel quantity is less than the first parallel quantity.

[0035] In conjunction with the first aspect, the present invention provides a sixth embodiment of the first aspect, specifically, further comprising:

[0036] During workflow execution, the operation logs of each computing unit are acquired; the operation logs include at least resource allocation records, data processing progress, abnormal exit information, and resource usage time sequence data of the computing unit.

[0037] The operation logs are indexed and stored according to the computing unit identity identifier and task identifier;

[0038] In response to a log query command, the corresponding runtime log is retrieved and displayed based on the computing unit's identity identifier or task identifier;

[0039] When an abnormal exit of a container instance is detected, the abnormal exit information is captured, and the retry process of the failed task is triggered based on the abnormal exit information.

[0040] In conjunction with the first aspect, the present invention provides a seventh embodiment of the first aspect, which specifically includes:

[0041] Obtain the actual resource usage data of each computing unit in the data processing stream during operation;

[0042] The actual resource usage data is compared with the current resource configuration parameters of the corresponding computing unit to calculate the resource utilization rate.

[0043] When the resource utilization rate is lower than a first threshold, a reduction suggestion to decrease the resource quota is generated; when the resource utilization rate is higher than a second threshold, an expansion suggestion to increase the resource quota is generated; wherein, the second threshold is greater than the first threshold;

[0044] After receiving confirmation of the reduction or expansion suggestion, update the resource configuration parameters of the corresponding computing unit and record the configuration change history.

[0045] Secondly, the present invention also provides a data processing system based on the separation of elastic scheduling and pre-labeling, comprising:

[0046] The image encapsulation storage module is used to acquire data processing logic, encapsulate the data processing logic into a computing unit image, and store it.

[0047] The computing unit generation module is used to obtain computing unit registration information and generate computing units based on the computing unit registration information; the computing unit registration information includes the storage address and running resource requirements of the computing unit image;

[0048] The data processing stream orchestration module is used to acquire orchestration instructions and generate a data processing stream containing pre-annotated calculation units according to the orchestration instructions; the data processing stream defines the execution order and dependencies of each calculation unit;

[0049] The scheduling and execution module is used to respond to the trigger command, pull the computing unit image corresponding to each computing unit from the image storage system according to the execution order and dependency relationship; and dynamically obtain runtime resources from the resource pool according to the running resource requirements to execute the instance of the computing unit image.

[0050] The pre-labeling processing module is used to automatically infer the input raw data in the runtime resources corresponding to the pre-labeling calculation unit, output the pre-labeled data, and store it in the data storage system.

[0051] The interactive annotation module is used to obtain the pre-annotated data from the data storage system and load it into the interactive interface, obtain operation data input based on the pre-annotated data, and generate target annotation result data.

[0052] Compared with the prior art, the beneficial effects of the present invention are:

[0053] This invention constructs a data processing flow that includes pre-annotation calculation units. During execution, these units automatically perform inference processing on the raw data and generate pre-annotated data, replacing the traditional purely manual frame-by-frame annotation method. Annotators only need to review and correct the pre-annotation results on the interactive interface, significantly reducing the amount of manual work. This is especially suitable for annotation scenarios involving massive amounts of video and point cloud data in autonomous driving, improving annotation efficiency by several times.

[0054] By combining automated reasoning and pre-annotation with manual verification, the efficient reasoning capabilities of AI models are leveraged while retaining the professional judgment of humans in complex boundary scenarios. This effectively reduces manual annotation errors caused by fatigue and subjective judgment differences, and improves the consistency and accuracy of the final target annotation results.

[0055] When triggered, this invention dynamically acquires runtime resources from a resource pool based on the declared runtime resource requirements in the registration information of each computing unit, rather than pre-allocating fixed resources. This mechanism supports efficient resource reuse and elastic scaling, and can allocate resources such as CPU, GPU, and memory on demand according to the actual load, avoiding resource idleness or contention and reducing system operation and maintenance costs. Attached Figure Description

[0056] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings, wherein:

[0057] Figure 1 This is a flowchart of a data processing method based on the separation of elastic scheduling and pre-labeling according to the present invention.

[0058] Figure 2 This is a block diagram of an electronic device used to implement embodiments of the present invention;

[0059] In the picture:

[0060] 100 - Electronic device, 101 - Computing unit, 102 - ROM, 103 - RAM, 104 - Bus, 105 - I / O interface, 106 - Input unit, 107 - Output unit, 108 - Storage unit, 109 - Communication unit. Detailed Implementation

[0061] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0062] like Figures 1-2 As shown, this invention provides a data processing method and system based on the separation of elastic scheduling and pre-labeling.

[0063] Example 1

[0064] The data processing method based on the separation of elastic scheduling and pre-labeling of the present invention can be executed by a data processing system based on the separation of elastic scheduling and pre-labeling. This system can be implemented in hardware and / or software and can be configured in an electronic device on a cloud server or a local server. The data processing method of the present invention is implemented based on a container orchestration platform, specifically, a Kubernetes cluster. Figure 1 As shown, the data processing method based on the separation of elastic scheduling and pre-labeling includes:

[0065] S100: Obtain the data processing logic, encapsulate the data processing logic into a computing unit image and store it.

[0066] In one specific implementation, the data processing logic is encapsulated as a computing unit image and stored, including:

[0067] S101: Obtain various types of data processing scripts, wherein the data processing scripts include at least pre-labeled model scripts and their dependent environments;

[0068] Specifically, the data processing system obtains the data processing scripts submitted by the developers. These scripts include format conversion scripts, frame extraction scripts, projection scripts, coordinate alignment scripts, and pre-labeled model scripts, along with their dependencies. For example, pre-labeled model scripts can be written based on the PyTorch or TensorFlow framework, and their dependencies include specific versions of the CUDA driver, the cuDNN library, and the corresponding Python packages.

[0069] S102: Each of the data processing scripts and their dependent environments is packaged into an independent container image using a pipeline tool;

[0070] Specifically, the system provides a standardized image building process through a pipeline tool. This pipeline tool can employ continuous integration tools such as Jenkins, GitLab CI, or GitHub Actions. After developers submit data processing scripts to the code repository, the pipeline tool automatically pulls the scripts and their dependencies, writes image building instructions using a Dockerfile, and packages the scripts and their dependencies into a container image. Different data processing logics are encapsulated into independent container images. For example, the point cloud motion compensation operator is encapsulated into one container image, the pre-annotation model operator into another, and the format conversion operator into yet another independent container image.

[0071] S103: Push each of the container images to the image repository for version management to obtain the computing unit image.

[0072] Specifically, each container image is tagged with a version number after it is built, such as using a semantic version number "v1.0.0" or a timestamp-based version identifier "20260115". After the container image is packaged, the system pushes the container image to an image repository. The image repository can use private or public image storage services such as Docker Hub, Harbor, or Alibaba Cloud Container Registry. The image repository stores and manages container images by version tag for on-demand retrieval.

[0073] It should be noted that, due to the significant differences in runtime environment requirements among different operators, some pre-annotated model images need to include a CUDA runtime environment and a large number of dependency packages, resulting in image sizes reaching tens of gigabytes. Versioning through unified pipeline tools and image repositories ensures the reproducibility and traceability of the operator environment.

[0074] S200: Obtain computing unit registration information, and generate a computing unit based on the computing unit registration information; the computing unit registration information includes the storage address and runtime resource requirements of the computing unit image;

[0075] In practice, the system provides an interface for the data management platform, through which operations and maintenance personnel input the registration information of operators. This registration information includes the operator name, operator description, image repository address, and version number. For example, when creating the "point cloud motion compensation" operator, operations and maintenance personnel would enter the image address and version number "v2.1.0".

[0076] In one specific implementation, obtaining computing unit registration information and generating computing units based on the computing unit registration information includes:

[0077] S201: Obtain the image repository address, version number, and resource configuration parameters of the computing unit; the resource configuration parameters include the number of CPU cores, memory capacity, whether a GPU is required, and the GPU memory size;

[0078] Specifically, operations and maintenance personnel also configure default resource parameters when creating operators. These resource parameters include the number of CPU cores, memory capacity, whether a GPU is required, and the size of the GPU memory. For example:

[0079] For the point cloud motion compensation operator, configure the CPU to have 4 cores and the memory capacity to be 8GB, and do not configure the GPU;

[0080] For pre-labeled model operators, configure the CPU to have 4 cores, the memory capacity to be 8GB, and the GPU memory to be 8GB;

[0081] For the format conversion operator, configure the CPU core count to 0.5 cores and the memory capacity to 512MB, and do not configure the GPU.

[0082] S202: Create a computing unit corresponding to the image repository address and version number on the data management platform, and associate it with the resource configuration parameters.

[0083] Specifically, after an operator is created, its registration information is stored in the platform's relational database. This registration information includes at least the operator's identifier, mirror address, version number, and resource configuration parameters. The operator's identifier is unique throughout the system and is used for reference during workflow orchestration and execution.

[0084] S300: Obtain orchestration instructions and generate a data processing stream containing pre-annotated computational units based on the orchestration instructions; the data processing stream defines the execution order and dependencies of each computational unit;

[0085] Specifically, operations and maintenance personnel use the workflow orchestration interface of the data management platform to select operators from the registered operator list and configure the execution order of each operator. For example, for point cloud data processing and pre-annotation scenarios, operations and maintenance personnel sequentially select the "point cloud vehicle motion compensation" operator, the "pre-annotation" operator, the "format conversion" operator, and the "frame extraction" operator, and set the serial execution order. The workflow orchestration interface also supports configuring data flow relationships. Operations and maintenance personnel can specify the output data of a preceding operator as the input data of a subsequent operator by dragging and dropping or configuring a form. For example, the output path of the "point cloud vehicle motion compensation" operator can be configured as the input path of the "pre-annotation" operator, and the output path of the "pre-annotation" operator can be configured as the input path of the "format conversion" operator. The workflow definition must contain at least one pre-annotation model operator. The pre-annotation model operator participates in the execution as a link in the data processing workflow. When the system executes the workflow, it will automatically call the pre-annotation model operator to complete offline inference during the preprocessing stage.

[0086] It's important to understand that a workflow is essentially structured configuration data that defines the execution order and data flow relationships between multiple operators. The system stores this configuration data in a database in YAML or JSON format. This structured configuration data includes mappings such as the identifier of each operator, its execution sequence position, the source of input data, and the target of output data. In one specific implementation, the system uses a directed acyclic graph topology to represent the execution order and dependencies between operators in the workflow.

[0087] In a preferred embodiment, generating a data processing stream containing pre-annotated calculation units according to the orchestration instructions includes:

[0088] S301: Obtain the identity identifiers of multiple computing units, their execution order parameters, and data flow parameters;

[0089] S302: Determine the dependencies between each computing unit based on the execution order parameters and data flow parameters; wherein, at least one pre-labeled computing unit is included, and the output data of the preceding computing unit is used as the input data of the subsequent computing unit;

[0090] S303: Generate the data processing stream, which includes the execution sequence of each computing unit and the input / output data mapping.

[0091] Specifically, by explicitly acquiring the identity, execution order, and data flow parameters of computational units, and accurately constructing dependencies between units accordingly, complex computational tasks can be decomposed into structured and manageable data processing flows. The introduction of pre-labeled computational units allows key data features or intermediate results to be identified and marked early in the process, providing clear processing anchors for subsequent units and effectively avoiding redundant computations and data ambiguity. Generating data processing flows containing execution sequences and input-output mappings not only achieves automated chaining and parallel scheduling optimization of computational tasks but also improves process traceability and maintainability. Users only need to adjust orchestration instructions to flexibly reconstruct computational logic without modifying the underlying code, significantly reducing development and maintenance costs.

[0092] S400: In response to the trigger command, according to the execution order and dependency relationship, pull the computing unit image corresponding to each computing unit from the image storage system;

[0093] Specifically, the system receives workflow trigger commands. These commands can originate from automatic system activation after data upload, scheduled task activation, or manual activation by maintenance personnel. For example, when a batch of raw point cloud and image data is uploaded to object storage, the system automatically detects the arrival of new data and triggers the execution of the corresponding workflow. The system parses the workflow definition based on the trigger command, obtaining the identifier of each operator in the workflow and its corresponding image repository address and version number. Following the execution order defined in the workflow, the system sequentially pulls the container images of each operator from the image repository. For example, for the "point cloud vehicle motion compensation" operator, the system pulls the "pointcloud-motion-compensation: v2.1.0" image from the image repository; for the "pre-labeling" operator, the system pulls the "pre-labeling: gpu-v1.3.0" image from the image repository. The system caches the pulled container images on the local nodes of the computing cluster. If the same image is used again in subsequent tasks, the system can directly load it from the local cache without repeated retrieval, thus reducing image retrieval time.

[0094] S500: Based on the runtime resource requirements, dynamically acquire runtime resources from the resource pool to execute an instance of the computing unit image; wherein, the pre-labeled computing unit performs automatic inference processing on the input raw data in its runtime resources, outputs pre-labeled data, and stores it in the data storage system;

[0095] In a preferred embodiment, the step of dynamically acquiring runtime resources from the resource pool according to the runtime resource requirements to execute an instance of the computing unit image includes:

[0096] Obtain the resource configuration parameters of each computing unit, including the number of CPU cores, memory capacity, and GPU memory size;

[0097] Specifically, the system obtains the resource parameters configured for each operator in the workflow, including the number of CPU cores, memory capacity, and GPU memory size. The system converts these resource parameters into Kubernetes ResourceRequirements objects, which serve as the basis for Pod resource requests and restrictions.

[0098] The resource configuration parameters are converted into container resource quotas. Based on the container resource quotas, runtime resources are dynamically requested in the computing cluster, and corresponding container instances are started. Among them, the container instance started by the pre-labeled computing unit loads the pre-labeled model, performs model inference on the input raw data, and outputs pre-labeled data. The container resource quota includes resource request quantity and resource limit quantity, wherein the resource limit quantity is greater than the resource request quantity. For GPU resources, only the resource limit quantity is set and the resource request quantity is not set.

[0099] Specifically, the system uses a differentiated strategy for configuring container resource quotas using requests and limits. Specifically, limits.cpu is set to twice requests.cpu, and limits.memory is set to twice requests.memory, to allow containers to obtain additional resources during short-term bursts of computation. For GPU resources, the system only sets limits and not requests, because GPU resources do not support overselling.

[0100] For example, for a point cloud motion compensation operator configured with a 4-core CPU and 8GB of memory, the system generates the following ResourceRequirements: requests.cpu=4, limits.cpu=8, requests.memory=8Gi, limits.memory=16Gi. For a pre-labeling operator configured with 8GB of video memory, the system additionally sets limits."nvidia.com / gpumem"=8Gi.

[0101] In practice, the system creates Job resources (for one-off tasks) or Deployment resources (for persistent tasks) through the Kubernetes API. The Kubernetes scheduler allocates Pods to compute nodes that meet the resource requirements based on the Pod's resource requests (requests value). For pre-labeled operators that require GPUs, the scheduler allocates them to nodes equipped with GPUs; for CPU-intensive operators that do not require GPUs, the scheduler allocates them to general-purpose compute nodes.

[0102] In one specific implementation, after the container instance of the pre-labeled model operator starts, it loads the pre-labeled model weight file from the container image. The container instance reads the raw point cloud and image data from the input data path and inputs the data into the pre-labeled model for inference. The pre-labeled model performs object detection on each frame of point cloud and image, generating pre-labeled results containing object detection boxes.

[0103] For example, the detection box data output by the pre-labeled model includes: center point x coordinate 1.7701, center point y coordinate -10.6179, center point z coordinate 0.7449, length dx is 4.6871, width dy is 2.0931, height dz is 1.5281, orientation angle rotZ is -1.5537, class label is "car", confidence score is 0.6635, and target ID is "632".

[0104] The pre-labeled data is stored independently in object storage in a structured data format, separated from the original data, and associated with it through timestamps;

[0105] Specifically, the pre-annotation model operator outputs the generated pre-annotation results in a structured data format, which can be JSON or Protocol Buffers. The system stores the pre-annotation result file in object storage, separately from the original point cloud and image data. The pre-annotation result file uses the frame name timestamp as its naming identifier, for example, "1777874976.json", and is associated with the corresponding original point cloud file "1777874976.pcd" and image file "1777874976.jpg" through the frame name timestamp.

[0106] Monitor the execution status of each container instance and automatically release the runtime resources it occupies after the task is completed.

[0107] Specifically, during the execution of a container instance, the system captures the container's standard output and standard error output through Kubernetes' logging mechanism and writes them to the log storage system. The system also collects container resource usage data, including CPU utilization, memory usage, and GPU memory usage, through monitoring components such as Prometheus. After the container instance completes execution, the system automatically deletes the corresponding Job resources or scales down the Deployment resources, releasing the compute node resources occupied by the Pod. These released resources are then available for use by other tasks in the cluster.

[0108] S600: Obtain the pre-labeled data from the data storage system and load it into the interactive interface, obtain operation data input based on the pre-labeled data, and generate target labeling result data.

[0109] In a preferred embodiment, the step of obtaining the pre-annotated data from the data storage system and loading it into the interactive interface, obtaining operation data input based on the pre-annotated data, and generating target annotation result data includes:

[0110] In response to the annotation task start command, the corresponding pre-annotation data is loaded from the object storage according to the timestamp of the current annotation frame; the pre-annotation data includes the position information, size information, orientation angle, category label and confidence level of the target detection box;

[0111] Specifically, annotators initiate annotation tasks through the annotation platform's client or web interface. The annotation platform loads the corresponding pre-annotation result file from object storage based on the frame name and timestamp of the current annotation frame. When an annotation task is opened, the platform automatically retrieves the pre-annotation result file associated with the current annotation frame. If a pre-annotation result file exists, the platform parses the JSON file and loads the pre-annotated data into the layers of the annotation interface. The annotation interface displays the original point cloud or image data, while simultaneously overlaying the pre-annotated detection boxes in the corresponding locations.

[0112] The pre-annotated data is displayed in the interactive interface of the annotation platform;

[0113] Specifically, annotators can see the position, size, orientation, and category label of the pre-annotated detection box on the annotation interface.

[0114] Obtain correction and supplementary operation data input by the annotation personnel based on the pre-annotated data;

[0115] Specifically, the labelers check the pre-labeling results, drag and adjust the position of inaccurate detection boxes, correct the incorrect category labels, and add new detection boxes for targets that were missed.

[0116] The pre-labeled data is updated based on the correction and supplementation operation data to generate the target labeling result data.

[0117] Specifically, after the annotators complete the correction and supplementation operations, the annotation platform updates the pre-annotated data based on the annotators' operation data, generating the final target annotation result data. The target annotation result data includes the corrected bounding box position information, size information, orientation angle, category label, etc. The annotation platform stores the target annotation result data in object storage or a database. After annotation is completed, the system performs post-processing operations on the target annotation result data. This post-processing includes converting the annotation results from the annotation platform format to a specific format required for model training (such as KITTI format or COCO format). After the format conversion is completed, the system delivers the converted training data to the model training process. After the annotation platform loads the pre-annotated result file, it displays a prompt message on the interface, informing the annotators that pre-annotated results already exist for the current frame. Annotators can choose to directly correct based on the pre-annotated results or choose to ignore the pre-annotated results and annotate manually.

[0118] It is important to understand that since the pre-annotation process is completed offline during the data preprocessing stage, the annotation platform does not need to call the online model inference service during the annotation process. Compared with existing technologies that require real-time requests to the model service during annotation, in this embodiment, the annotation platform only needs to load a lightweight structured JSON file to obtain the pre-annotation results. It does not need to deploy and maintain GPU inference services, nor does it need to bear the performance pressure of the model service during concurrent annotation.

[0119] In one specific embodiment, historical resource usage data of each computing unit is obtained; the historical resource usage data includes actual CPU usage, actual memory usage, and actual GPU memory usage.

[0120] Specifically, the system acquires historical resource usage data generated by each computing unit during the execution of data processing tasks. This historical resource usage data is collected by monitoring components deployed in the computing cluster. For example, the system deploys the Prometheus monitoring service in a Kubernetes cluster, using collectors such as cAdvisor and node-exporter to periodically collect time-series data on CPU usage, memory usage, and GPU memory usage from each Pod and computing node.

[0121] After each computational unit completes its execution, the system persistently stores the resource usage data for that execution period in a time-series database (such as PrometheusTSDB or InfluxDB). This resource usage data includes, but is not limited to: execution start time, execution end time, average CPU utilization, peak CPU utilization, average memory usage, peak memory usage, average GPU memory usage, and peak GPU memory usage. The system organizes and indexes this data according to operator identifiers and execution timestamps.

[0122] In one specific implementation, after collecting sufficient historical data, the system initiates the resource recommendation analysis process. The system performs the analysis through the VerticalPodAutoscaler component in the Kubernetes cluster. VerticalPodAutoscaler runs in Off mode, retrieving historical resource usage data for Pods from KubernetesMetricsServer or Prometheus, and performing statistical analysis on the data by percentile.

[0123] It's important to note that VerticalPodAutoscaler runs in Off mode, only performing data analysis and outputting recommended values; it will not automatically trigger Pod restarts or resource configuration changes. This mode ensures that the analysis process does not affect running business tasks.

[0124] The historical resource usage data is analyzed to generate resource recommendation values;

[0125] In one specific implementation, the VerticalPodAutoscaler analysis process is as follows: It collects resource usage data for the specified Pod over the past 8 days; calculates the 90th percentile for both CPU and memory usage as the target value; calculates the 50th percentile as the lower limit; and calculates the 95th percentile as the upper limit. The system outputs the analysis results in YAML format.

[0126] For example, for a computing unit performing point cloud motion compensation, the recommended resource values ​​output by VerticalPodAutoscaler are: target: {cpu: 4.5, memory: 12Gi}, lowerBound: {cpu: 3.2, memory: 8Gi}, upperBound: {cpu: 6.8, memory: 16Gi}. This output indicates that the recommended configuration is 4.5 CPU cores and 12GB of memory, with resource usage fluctuating between 3.2 and 6.8 cores and 8GB and 16GB.

[0127] For GPU memory resources, since VerticalPodAutoscaler does not support recommended analysis for GPU resource types, the system employs different analysis methods. In one specific implementation, the system collects GPU memory usage data for each GPU container instance during execution via Prometheus. The algorithm team or operations personnel obtain historical peak GPU memory data from the monitoring panel. Based on the peak memory usage data from multiple historical executions, the maximum value is taken and a preset redundancy (e.g., 10%) is added as the recommended GPU memory value. For example, if the highest peak GPU memory usage for a certain pre-labeled model operator in all executions is 7.2GB, then the system or operations personnel will determine the recommended GPU memory value to be 8GB.

[0128] The resource configuration parameters of the corresponding computing unit are updated based on the resource recommendation value, and the updated resource configuration parameters take effect when the next data processing stream is triggered.

[0129] In one specific implementation, the system pushes the recommended CPU and memory values, as well as the recommended GPU memory values, output by VerticalPodAutoscaler to the operations and maintenance management interface for display. The operations and maintenance management interface displays the name of each computing unit, its current resource configuration, recommended resource values, and recommended time in a list format. Operations and maintenance personnel can review the recommended values.

[0130] In one specific implementation, after the operations and maintenance personnel confirm the recommended resource values ​​through the operations and maintenance management interface, they trigger a resource configuration update operation. The system receives the update instruction and modifies the resource configuration parameters of the corresponding computing unit to the recommended values. For example, the number of CPU cores in the point cloud motion compensation operator is updated from 4 cores to 4.5 cores, and the memory capacity is updated from 8GB to 12GB.

[0131] It's important to understand that updated resource configuration parameters do not immediately take effect on running tasks. There may already be running Pods using older resource configurations. Updated resource configuration parameters are stored in the platform database and only apply to newly triggered tasks.

[0132] In one specific implementation, when the system receives the trigger instruction for the next data processing stream, the system reads the latest resource configuration parameters of the computing unit from the database when constructing the Pod resource request. The updated parameter values ​​are used as the basis for the values ​​of requests and limits in KubernetesResourceRequirements, so that node resources are requested according to the new resource quota when scheduling the Pod.

[0133] In one specific implementation, the system also supports operations and maintenance personnel to manually adjust resource configuration parameters, bypassing recommended values. Personnel can directly input custom CPU core count, memory capacity, and GPU memory size in the operator configuration interface; the system stores these custom values ​​and uses them the next time a task is triggered.

[0134] It should be noted that the dynamic update mechanism for resource configuration parameters is based on the historical actual usage of each computing unit, rather than relying on preset static configuration rules. Therefore, resource configuration can be gradually adjusted to a level that adapts to the current actual needs as the data scale or model complexity changes, avoiding resource shortages or waste caused by changes in data characteristics.

[0135] In a preferred embodiment, the estimated execution time of a single task corresponding to each computing unit is obtained; the estimated execution time is compared with a preset threshold, and the number of parallel instances of the corresponding computing unit is automatically adjusted according to the comparison result; wherein, if the estimated execution time is less than a first threshold, a first parallel quantity is configured; if the estimated execution time is greater than a second threshold, a second parallel quantity is configured, wherein the second parallel quantity is less than the first parallel quantity.

[0136] In one specific implementation, the system acquires historical time data for each computing unit executing a single data processing task. After each computing unit completes its execution, the system records the start and end times of that execution and calculates the execution time. The execution time is then associated with the corresponding computing unit's identifier and data scale information and stored in the database. For example, a point cloud motion compensation task processes 3000 frames of point cloud data, with an execution time of 3 minutes and 20 seconds. The system associates this time record with the operator's historical execution record table.

[0137] In one specific implementation, once the system has accumulated a certain number of historical execution records (more than 10 execution records under the same data scale level for the same operator), the system calculates the estimated execution time for a single task of that operator. Specifically, the system takes the average of the execution times from multiple executions as the estimated execution time, or takes the median of the execution times from multiple executions as the estimated execution time, in order to eliminate statistical biases caused by occasional execution anomalies.

[0138] For example, a pre-labeling operator is executed 20 times consecutively for a data scale of 3000 frames. The execution times are 2 minutes 50 seconds, 3 minutes 10 seconds, 2 minutes 55 seconds, 3 minutes 05 seconds, ..., 3 minutes 00 seconds. The system calculates the average value as 3 minutes 02 seconds, and this average value is determined as the estimated execution time of a single task under this data scale.

[0139] In one specific implementation, after determining the estimated execution time, the system compares the estimated execution time with a preset threshold. The preset threshold includes a first threshold and a second threshold, wherein the first threshold is less than the second threshold. For example, the first threshold is set to 1 minute, and the second threshold is set to 5 minutes.

[0140] In practice, the system automatically adjusts the number of parallel instances for the corresponding computing unit based on the comparison results. The number of parallel instances (job_count) represents the number of container instances that can run simultaneously within the same computing unit at the same time. This parameter is decoupled from the resource requests of each instance (number of CPU cores, memory capacity, GPU memory). The former controls the number of concurrently executing instances, while the latter controls the resource quota occupied by each instance.

[0141] If the estimated execution time is less than the first threshold, it indicates that the single task processing speed is relatively fast, and the system configures a first parallel quantity. The first parallel quantity is an integer greater than 1. For example, when the estimated execution time is 45 seconds (less than 1 minute), the system sets the number of parallel instances to 4, meaning that the system simultaneously starts 4 container instances to process 4 independent sets of data in parallel.

[0142] If the estimated execution time exceeds the second threshold, it indicates that the single-task processing speed is slow, possibly involving intensive computation or a large amount of data I / O. In this case, the system is configured with a second parallel processing level. The second parallel processing level is less than the first parallel processing level.

[0143] In one specific implementation, the second parallel quantity is 1, meaning that the system starts only one container instance at a time to process data serially, avoiding a decrease in overall throughput due to multiple instances competing for limited cluster resources (such as disk I / O, network bandwidth, or GPU memory).

[0144] For example, when the estimated execution time is 6 minutes and 30 seconds (greater than 5 minutes), the system sets the number of parallel instances to 1, and the data is processed sequentially, with only one instance's resources being used at any given time.

[0145] In one specific implementation, if the estimated execution time falls between a first threshold and a second threshold, the system configures a third number of parallel instances. This third number of parallel instances is between the first and second number of parallel instances. For example, if the estimated execution time is 3 minutes, falling between 1 and 5 minutes, the system sets the number of parallel instances to 2.

[0146] In one specific implementation, after determining the number of parallel instances, the system stores this number as a configuration parameter in the compute unit configuration record in the database. When the system receives the next workflow trigger instruction, it creates the corresponding number of Pod replicas or Job instances through the Kubernetes API based on the number of parallel instances for that compute unit.

[0147] For example, if the number of parallel instances of a format conversion operator is configured to 4, the system will create 4 parallel Kubernetes Jobs when triggering workflow execution, with each Job processing an independent subset of data; if the number of parallel instances is configured to 1, the system will create only 1 Job to process all data serially.

[0148] In one specific implementation, the system also dynamically adjusts the number of parallel instances based on changes in the estimated execution time. When the actual execution time of the same computing unit changes in subsequent executions, the system recalculates the estimated execution time and re-compares it with the threshold. For example, if a computing unit's estimated execution time was originally 40 seconds (less than 1 minute) and the number of parallel instances was 4; subsequently, due to the increase in data size, the actual execution time gradually increased to 1 minute and 20 seconds. The system then recalculates the estimated execution time to 1 minute and 15 seconds (between 1 minute and 5 minutes) and adjusts the number of parallel instances from 4 to 2 to balance parallel efficiency and resource contention.

[0149] In a preferred embodiment, during workflow execution, the operation logs of each computing unit are acquired; the operation logs include at least resource allocation records, data processing progress, abnormal exit information, and resource usage time sequence data of the computing units;

[0150] Specifically, the system captures and records the runtime logs of each computing unit during workflow execution. These runtime logs originate from the standard output stream (stdout) and standard error stream (stderr) of the container instance. Upon startup, the container instance's standard output and standard error are redirected to a log collection agent (such as Fluentd or Filebeat), which collects the data and sends it to the log storage system. The runtime logs include at least resource allocation records, data processing progress, abnormal exit information, and time-series data on resource usage for the computing unit. Resource allocation records are generated during container instance startup and include: the name of the compute node allocated to the Pod by the Kubernetes scheduler, the number of CPU cores allocated, the memory capacity, and the GPU memory size. For example, a resource allocation record can be represented as: "Pod point cloud motion compensation-abc123 is scheduled to node-gpu-05, allocated resources requests: cpu=4, memory=8Gi, limits: cpu=8, memory=16Gi, gpu=8Gi".

[0151] Data processing progress logs are output at preset time intervals or according to the data processing ratio during the execution of the computing unit. For example, a pre-labeling operator processing 3000 frames of point cloud data outputs a progress log every 500 frames processed: "500 / 3000 frames processed, progress 16.7%", "1000 / 3000 frames processed, progress 33.3%", and so on. The progress logs enable operations and maintenance personnel and developers to understand the task execution status in real time, making it easier to determine whether the task is progressing normally or experiencing delays.

[0152] Abnormal exit information is generated when a container instance terminates due to an error. Abnormal exit information includes: exit code, error stack trace, and the time the error occurred. For example, when a pre-annotated model container abnormally exits due to insufficient GPU memory, an error log is output, and the exit code is 1.

[0153] Resource usage time-series data provides the data foundation for resource recommendation value analysis. Specifically, resource usage time-series data includes sampled data on CPU utilization, memory usage, and GPU memory usage over time, collected by monitoring components deployed in the computing cluster at preset sampling intervals (e.g., every 30 seconds).

[0154] The operation logs are indexed and stored according to the computing unit identity identifier and task identifier;

[0155] Specifically, the system uniformly writes the collected operation logs to a log storage system. This log storage system can be a log storage and analysis platform with full-text search capabilities, such as Elasticsearch, Loki, or Alibaba Cloud Log Service (SLS). During log writing, the system adds an index field to each log entry. This index field includes at least: a computational unit identifier (e.g., operator name), a task identifier (e.g., workflow instance ID), a container instance identifier (e.g., Pod name), a log generation timestamp, and a log level (e.g., INFO, WARN, ERROR).

[0156] In response to a log query command, the corresponding runtime log is retrieved and displayed based on the computing unit's identity identifier or task identifier;

[0157] Specifically, the system responds to log query commands initiated by operations and maintenance personnel or developers through the operations and maintenance management interface. The operations and maintenance management interface provides a query form where users can input query conditions such as computing unit identifiers (e.g., selecting or entering operator names), task identifiers (e.g., workflow instance IDs), time ranges, and log levels. Based on the user-input query conditions, the system constructs a structured query statement (such as Elasticsearch's QueryDSL or SQL statements) and sends a retrieval request to the log storage system.

[0158] When an abnormal exit of a container instance is detected, the abnormal exit information is captured, and the retry process of the failed task is triggered based on the abnormal exit information.

[0159] Specifically, the system detects the exit status of container instances through Kubernetes' Pod status monitoring mechanism. When a Pod status of Failed or Error is detected, the system captures the abnormal exit information and automatically triggers a retry process for the failed task. The retry process includes: recreating the Pod instance and submitting it to the Kubernetes scheduler according to a preset retry strategy (e.g., a maximum of 3 retries with a 30-second interval between each retry). If the retry is successful, the system records the task recovery log; if the retry count reaches the limit and the task still fails, the system generates an alarm notification and marks the task as a final failure status for operations and maintenance personnel to intervene and investigate.

[0160] In a preferred embodiment, the actual resource usage data of each computing unit in the data processing stream during operation is obtained; the actual resource usage data is compared with the current resource configuration parameters of the corresponding computing unit to calculate the resource utilization rate;

[0161] When the resource utilization rate is lower than a first threshold, a reduction suggestion to decrease the resource quota is generated; when the resource utilization rate is higher than a second threshold, an expansion suggestion to increase the resource quota is generated; wherein, the second threshold is greater than the first threshold; after receiving confirmation of the reduction suggestion or expansion suggestion, the resource configuration parameters of the corresponding computing unit are updated, and the configuration change history is recorded.

[0162] Specifically, the system acquires the actual resource usage data of each computing unit in the data processing stream during operation. This resource usage data is collected by monitoring components deployed in the computing cluster. In a Kubernetes cluster, the system uses MetricsServer or Prometheus in conjunction with cAdvisor to collect the CPU and memory usage of each Pod; for GPU memory usage, it uses NVIDIA DCGM or Prometheus's nvidia-gpu-exporter to collect the memory usage of each GPU container.

[0163] In one specific implementation, after each computational unit completes its execution, the system retrieves resource usage data from the monitoring system for that execution period. The retrieval method includes querying time-series data within a specified time range via the Prometheus API to obtain peak CPU utilization, peak memory usage, and peak GPU memory usage. For example, for the 20260115-001st task execution of a point cloud motion compensation operator, the system queries Prometheus and finds a peak CPU utilization of 3.8 cores and a peak memory usage of 6.5GB.

[0164] In one specific implementation, the system obtains the resource configuration parameters used by the computing unit during execution. These resource configuration parameters are read from the computing unit configuration records in the database, including the number of CPU cores and memory capacity. For example, the current configuration of the point cloud motion compensation operator is 4 CPU cores and 8GB of memory. The system compares the actual resource usage data with the current resource configuration parameters to calculate the resource utilization rate.

[0165] More specifically, CPU utilization = peak actual CPU usage / CPU requests. Following the example above, CPU utilization = 3.8 cores / 4 cores = 95%. Memory utilization = peak actual memory usage / memory requests, i.e., 6.5GB / 8GB = 81.25%. For GPU memory utilization, since the GPU only sets limits and not requests, GPU memory utilization = peak actual GPU memory usage / GPU memory limit. For example, if a pre-labeled operator configures GPU memory to 8GB and the peak actual usage is 6.2GB, then the GPU memory utilization is 77.5%.

[0166] In one specific implementation, the system compares the calculated resource utilization rate with a preset threshold. The preset threshold includes a first threshold and a second threshold, where the second threshold is greater than the first threshold. For example, the first threshold is set to 30%, and the second threshold is set to 80%. The first threshold is used to determine whether there is resource waste (over-configuration), and the second threshold is used to determine whether there is resource insufficiency (under-configuration).

[0167] When resource utilization falls below the first threshold, the system generates a reduction suggestion to lower resource quotas. Using the example above where CPU utilization is 95%, this value is above the second threshold, and no reduction suggestion is triggered.

[0168] When resource utilization exceeds the second threshold, the system generates an expansion suggestion to increase resource quotas. In the scenario described above, where CPU utilization is 95%, this value is above 80%, so the system generates an expansion suggestion to increase the number of CPU cores from 4 to 5, and displays this suggestion in the operations and maintenance management interface.

[0169] If the resource utilization rate is between the first and second thresholds, the system will not generate adjustment suggestions and will maintain the current resource configuration.

[0170] Example 2

[0171] A data processing system based on elastic scheduling and pre-labeling separation includes: an image encapsulation and storage module for acquiring data processing logic, encapsulating the data processing logic into computing unit images, and storing them; a computing unit generation module for acquiring computing unit registration information and generating computing units based on the computing unit registration information; the computing unit registration information includes the storage address and runtime resource requirements of the computing unit images; a data processing stream orchestration module for acquiring orchestration instructions and generating a data processing stream containing pre-labeled computing units based on the orchestration instructions; the data processing stream defines the execution order and dependencies of each computing unit; and a scheduling execution module for... In response to a trigger command, the system retrieves the computing unit images corresponding to each computing unit from the image storage system according to the execution order and dependencies; and dynamically acquires runtime resources from the resource pool according to the runtime resource requirements to execute instances of the computing unit images; a pre-annotation processing module is used to automatically infer the input raw data in the runtime resources corresponding to the pre-annotated computing unit, output pre-annotated data, and store it in the data storage system; an interactive annotation module is used to obtain the pre-annotated data from the data storage system and load it into the interactive interface, obtain operation data input based on the pre-annotated data, and generate target annotation result data.

[0172] Example 3

[0173] According to embodiments of the present invention, the present invention also provides an electronic device, a readable storage medium, and a computer program product.

[0174] Figure 2 A schematic block diagram of an example electronic device 100 that can be used to implement embodiments of the present invention is shown. Electronic device 100 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic device 100 may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their links and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0175] like Figure 2As shown, the electronic device 100 includes a computing unit 101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 102 or a computer program loaded from a storage unit 108 into a random access memory (RAM) 103. The RAM 103 may also store various programs and data required for the operation of the device 100. The computing unit 101, ROM 102, and RAM 103 are interconnected via a bus 104. An input / output (I / O) interface 105 is also linked to the bus 104.

[0176] Multiple components in electronic device 100 are linked to I / O interface 105, including: input unit 106, such as keyboard, mouse, etc.; output unit 107, such as various types of displays, speakers, etc.; storage unit 108, such as disk, optical disk, etc.; and communication unit 109, such as network card, modem, wireless transceiver, etc. Communication unit 109 allows electronic device 100 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0177] The computing unit 101 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 101 performs the various methods and processes described above, such as a data processing method based on flexible scheduling and pre-labeling separation. For example, in some embodiments, a data processing method based on flexible scheduling and pre-labeling separation can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 108. In some embodiments, part or all of the computer program can be loaded and / or installed on device 100 via ROM 102 and / or communication unit 109. When the computer program is loaded into RAM 103 and executed by the computing unit 101, one or more steps of a data processing method based on flexible scheduling and pre-labeling separation described above can be performed. Alternatively, in other embodiments, computing unit 101 may be configured by any other suitable means (e.g., by means of firmware) to perform a data processing method based on the separation of flexible scheduling and pre-labeling.

[0178] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0179] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0180] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical links based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0181] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0182] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0183] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0184] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0185] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Therefore, any modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A data processing method based on the separation of elastic scheduling and pre-labeling, characterized in that, include: Obtain the data processing logic, encapsulate the data processing logic into a computing unit image, and store it; Obtain computing unit registration information, and generate computing units based on the computing unit registration information; The computing unit registration information includes the storage address and runtime resource requirements of the computing unit image; Obtain orchestration instructions, and generate a data processing flow containing pre-annotated computational units based on the orchestration instructions; the data processing flow defines the execution order and dependencies of each computational unit; In response to the trigger command, the computing unit images corresponding to each computing unit are retrieved from the image storage system according to the execution order and dependencies. Based on the aforementioned runtime resource requirements, runtime resources are dynamically acquired from the resource pool to execute instances of the computing unit image; wherein, the pre-labeled computing unit performs automatic inference processing on the input raw data within its runtime resources, outputs pre-labeled data, and stores it in the data storage system; The pre-labeled data is obtained from the data storage system and loaded into the interactive interface. Operation data based on the pre-labeled data is obtained and target labeling result data is generated.

2. The data processing method based on the separation of elastic scheduling and pre-labeling according to claim 1, characterized in that, The step of obtaining computing unit registration information and generating computing units based on the computing unit registration information includes: Obtain the image repository address, version number, and resource configuration parameters of the computing unit; the resource configuration parameters include the number of CPU cores, memory capacity, whether a GPU is required, and the size of the GPU memory. Create a computing unit on the data management platform that corresponds to the image repository address and version number, and associate it with the resource configuration parameters.

3. The data processing method based on the separation of elastic scheduling and pre-labeling according to claim 1, characterized in that, The step of dynamically acquiring runtime resources from the resource pool based on the runtime resource requirements to execute an instance of the computing unit image includes: Obtain the resource configuration parameters of each computing unit, including the number of CPU cores, memory capacity, and GPU memory size; The resource configuration parameters are converted into container resource quotas. Based on the container resource quotas, runtime resources are dynamically requested in the computing cluster, and the corresponding container instances are started. Here, is the container instance started by the pre-labeling calculation unit, which loads the pre-labeling model, performs model inference on the input raw data, and outputs pre-labeled data; The pre-labeled data is stored independently in object storage in a structured data format, separated from the original data, and associated with it through timestamps; Monitor the execution status of each container instance and automatically release the runtime resources it occupies after the task is completed.

4. The data processing method based on the separation of elastic scheduling and pre-labeling according to claim 1, characterized in that, The steps of obtaining the pre-annotated data from the data storage system and loading it into the interactive interface, obtaining operation data input based on the pre-annotated data, and generating target annotation result data include: In response to the annotation task start command, the corresponding pre-annotation data is loaded from the object storage according to the timestamp of the current annotation frame; the pre-annotation data includes the position information, size information, orientation angle, category label and confidence level of the target detection box; The pre-annotated data is displayed in the interactive interface of the annotation platform; Obtain correction and supplementary operation data input by the annotation personnel based on the pre-annotated data; The pre-labeled data is updated based on the correction and supplementation operation data to generate the target labeling result data.

5. The data processing method based on the separation of elastic scheduling and pre-labeling according to claim 1, characterized in that, Also includes: Obtain historical resource usage data for each computing unit; the historical resource usage data includes actual CPU usage, actual memory usage, and actual GPU memory usage; The historical resource usage data is analyzed to generate resource recommendation values; The resource configuration parameters of the corresponding computing unit are updated based on the resource recommendation value, and the updated resource configuration parameters take effect when the next data processing stream is triggered.

6. The data processing method based on the separation of elastic scheduling and pre-labeling according to claim 1, characterized in that, Also includes: Obtain the estimated execution time of each task corresponding to each computing unit; The estimated execution time is compared with a preset threshold, and the number of parallel instances of the corresponding computing unit is automatically adjusted according to the comparison result; wherein, if the estimated execution time is less than the first threshold, a first parallel quantity is configured; if the estimated execution time is greater than the second threshold, a second parallel quantity is configured, and the second parallel quantity is less than the first parallel quantity.

7. The data processing method based on the separation of elastic scheduling and pre-labeling according to claim 1, characterized in that, Also includes: During workflow execution, the operation logs of each computing unit are acquired; the operation logs include at least resource allocation records, data processing progress, abnormal exit information, and resource usage time sequence data of the computing unit. The operation logs are indexed and stored according to the computing unit identity identifier and task identifier; In response to a log query command, the corresponding runtime log is retrieved and displayed based on the computing unit's identity identifier or task identifier; When an abnormal exit of a container instance is detected, the abnormal exit information is captured, and the retry process of the failed task is triggered based on the abnormal exit information.

8. The data processing method based on the separation of elastic scheduling and pre-labeling according to claim 1, characterized in that, Also includes: Obtain the actual resource usage data of each computing unit in the data processing stream during operation; The actual resource usage data is compared with the current resource configuration parameters of the corresponding computing unit to calculate the resource utilization rate. When the resource utilization rate is lower than a first threshold, a reduction suggestion to decrease the resource quota is generated; when the resource utilization rate is higher than a second threshold, an expansion suggestion to increase the resource quota is generated; wherein, the second threshold is greater than the first threshold; After receiving confirmation of the reduction or expansion suggestion, update the resource configuration parameters of the corresponding computing unit and record the configuration change history.

9. A data processing system based on the separation of elastic scheduling and pre-labeling, characterized in that, include: The image encapsulation storage module is used to acquire data processing logic, encapsulate the data processing logic into a computing unit image, and store it. A computing unit generation module is used to obtain computing unit registration information and generate computing units based on the computing unit registration information. The computing unit registration information includes the storage address and runtime resource requirements of the computing unit image; The data processing stream orchestration module is used to acquire orchestration instructions and generate a data processing stream containing pre-annotated calculation units according to the orchestration instructions; the data processing stream defines the execution order and dependencies of each calculation unit; The scheduling and execution module is used to respond to the trigger command and, according to the execution order and dependency relationship, pull the computing unit image corresponding to each computing unit from the image storage system; And based on the aforementioned runtime resource requirements, dynamically acquire runtime resources from the resource pool to execute instances of the computing unit image; The pre-labeling processing module is used to automatically infer the input raw data in the runtime resources corresponding to the pre-labeling calculation unit, output the pre-labeled data, and store it in the data storage system. The interactive annotation module is used to obtain the pre-annotated data from the data storage system and load it into the interactive interface, obtain operation data input based on the pre-annotated data, and generate target annotation result data.

10. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the data processing method according to any one of claims 1-7.

Images