A data production method, device, equipment and medium

Abstract

The present application relates to the technical field of automatic driving data processing, and discloses a data production method, device, equipment and medium, the method comprising: obtaining original perception data; obtaining a corresponding task configuration template according to an automatic driving training task; binding the original perception data and the task configuration template to generate a current task instance; based on the type of the current task instance, the current task instance is issued to a corresponding target ordinary POD or target resident POD for execution; and obtaining training data obtained after execution of the current task instance is completed. The present application improves the efficiency and accuracy of data production.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving data processing technology, specifically to a data production method, apparatus, equipment, and medium. Background Technology

[0002] As the iteration speed of autonomous driving models continues to accelerate, the industry is placing higher demands on the production capacity, processing efficiency, and customization capabilities of training data. Currently, multiple models, including perception, prediction, and planning models, are iterating simultaneously, with different models and versions corresponding to differentiated data pipelines. These pipelines include various processing operators such as point cloud filtering, image enhancement, and multimodal fusion, resulting in a large number of heterogeneous data tasks and multiple versions of datasets. Related data processing solutions can only complete basic driving data collection, scene extraction, and standardized preprocessing. They suffer from shortcomings such as inefficient resource scheduling and complex task configuration, making them difficult to adapt to multi-model parallel development scenarios and leading to reduced data production efficiency. Summary of the Invention

[0003] This invention provides a data production method, apparatus, equipment, and medium to solve the problem of low efficiency in training data production.

[0004] In a first aspect, the present invention provides a data production method applied to a production platform with schedulable cloud resources. The method includes: acquiring raw perception data; acquiring a corresponding task configuration template based on an autonomous driving training task, wherein the task configuration template is a template pre-configured and stored based on the user's data production requirements, and the production requirements correspond to the autonomous driving training task; binding the raw perception data and the task configuration template to generate a current task instance; distributing the current task instance to a corresponding target ordinary POD or target resident POD for execution based on the type of the current task instance, wherein the target ordinary POD is used to create when the task is distributed and release cloud resources when the task ends, and the target resident POD is used to create when the task is distributed for the first time and release cloud resources after receiving a stop command and all internally running tasks are completed; and acquiring training data obtained after the current task instance is executed.

[0005] Based on the aforementioned technical methods, task templates matching the training requirements of autonomous driving are pre-configured. Raw perception data is bound to these templates to generate standardized task instances, reducing the tedious manual configuration steps required for each production cycle. Furthermore, tasks are executed using two types of containers: ordinary PODs and resident PODs. Ordinary PODs release resources elastically with task start and stop, avoiding long-term idle resource occupation; resident PODs are reused for tasks with the same attributes, eliminating the overhead of repeatedly creating and destroying containers. Simultaneously, standardized training data is output uniformly, solving the problems of cumbersome data pipeline configuration and inefficient resource scheduling in multi-model parallel development. The entire process achieves end-to-end automation of data production, balancing elastic task scheduling and resource reuse, significantly improving the overall throughput efficiency of autonomous driving training data production and reducing cloud resource idleness.

[0006] In some optional implementations, the steps of configuring the task configuration template include: configuring data processing version information, resource and environment configuration information, processing link parameters and task management identifier. The data processing version information is used to specify the operator version used for data filtering and processing. The resource and environment configuration information is used to define the hardware resource information for task execution. The processing link parameters are used to represent the execution parameters required by the data processing operators. The task management identifier is used to distinguish task types and mark task priorities.

[0007] Based on the aforementioned technical methods, task templates are standardized across multiple dimensions, defining four core information categories: operator version, hardware resources, link parameters, and task management identifiers. Operator versions unify data processing standards, ensuring consistency across multiple batches of data output; resource configuration pre-locks hardware specifications such as CPU / GPU and memory, eliminating the need for repeated resource parameter verification during scheduling; link parameters solidify operator execution logic, reducing repetitive manual configuration; and task identifiers differentiate task types and priorities, providing a basis for subsequent hierarchical scheduling. This unified template configuration simplifies the construction of multi-version data pipelines, reduces the probability of errors in manual configuration, and supports the rapid creation of massive amounts of differentiated autonomous driving tasks.

[0008] In some optional implementations, the raw sensing data and the task configuration template are bound together to generate the current task instance, including: if the raw sensing data is a single data item, the raw sensing data and the task configuration template are directly bound together to generate the current task instance; if the raw sensing data is batch data, the raw sensing data is divided into multiple independent data units; and each data unit is bound to the task configuration template to generate multiple corresponding current task instances.

[0009] Based on the aforementioned technical methods, templates are directly bound to single data entries to quickly generate tasks; batch data is automatically split into independent data units, generating independent task instances one by one. After batch splitting, the data volume of a single task is controllable, avoiding scheduling blockages caused by running a single instance of extremely large batch tasks; the split tasks can be distributed in parallel to multiple nodes in the cluster for processing, making full use of distributed computing power. There is no need for manual splitting of batch data, reducing the cost of preprocessing large batches of perceived data, balancing the load on single tasks, and improving the cluster's concurrent processing capabilities.

[0010] In some optional implementations, the current task instance is deployed to the corresponding target ordinary POD or target persistent POD for execution based on its type. This includes: if the current task instance is defined as an ordinary task, then when idle cloud resources meet the availability conditions of the current task instance, a target ordinary POD is created for the current task instance; the current task instance is deployed to the target ordinary POD for execution, and the target ordinary POD is released when the current task instance finishes execution; if the current task instance is defined as a persistent task, it is determined whether there is a first task instance deployed and executed before the current task instance, and the first task instance has the same attribute information as the current task instance; if not, then when idle cloud resources meet the availability conditions of the current task instance, a first target persistent POD is created for the current task instance, and the current task instance is deployed to the first target persistent POD for execution; if so, then a second target persistent POD that has run the first task instance is found, and when the second target persistent POD has the resource conditions to receive the current task instance, the current task instance is deployed to the second target persistent POD for execution.

[0011] Based on the aforementioned technical methods, ordinary tasks create new PODs on demand and release them immediately after execution, suitable for low-frequency, differentiated temporary data tasks. Permanent tasks with the same attributes reuse existing permanent PODs, creating new containers only when no matching instance exists, reducing the startup and destruction time caused by frequent container creation and destruction. Simultaneously, attribute matching filters for reusing target permanent PODs, controlling the centralized execution of tasks with the same template and reducing resource consumption during program switching. This two-tiered task scheduling approach balances the elastic resource needs of temporary tasks with the resource reuse needs of high-frequency, similar tasks, significantly reducing cluster resource overhead.

[0012] In some optional implementations, the steps of executing internal tasks through a target ordinary POD or a target resident POD include: splitting all task instances within the current POD into multiple execution nodes according to the smallest execution unit; and selecting multiple execution nodes that do not conflict with each other to execute concurrently within the current POD according to the execution order of each execution node and the type of hardware resources used.

[0013] Based on the aforementioned technical methods, tasks within a POD are broken down into the smallest execution nodes. Nodes without resource conflicts within the same task and between different tasks execute in parallel, fully leveraging the potential of multiple hardware resources (CPU and GPU) within a single container. Multiple task nodes are interleaved in parallel, avoiding idle single hardware channels. There is no need to wait for all preceding steps to complete before executing subsequent processes, shortening the overall processing time of a single task and maximizing the utilization of hardware within ordinary and resident PODs. This addresses the shortcomings of traditional serial processing modes, such as wasted container computing power and high processing latency.

[0014] In some optional implementations, when the idle cloud resources meet the availability conditions of the current task instance, the step of creating a target ordinary POD or a target resident POD for the current task instance includes: determining the number of tasks that can be issued in this period based on the current idle cloud resources; sorting the task instances based on the priority corresponding to the current task instance and the priorities corresponding to other task instances in this period, and determining whether the current task instance is within the number of tasks; if the current task instance is within the number of tasks, then creating a target ordinary POD or a target resident POD for the current task instance and updating the current idle cloud resources.

[0015] Based on the aforementioned technical methods, the upper limit of tasks that can be deployed in a single cycle is determined according to the total amount of idle resources. Tasks are then prioritized and deployed based on their priority. When resources are insufficient, low-priority tasks are postponed. Before scheduling, the remaining computing power of the cluster is assessed to avoid task queuing and resource contention caused by over-deploying resources. Priority ranking ensures that high-value autonomous driving training tasks have priority access to computing power. This achieves refined quota management of cluster resources, balances concurrent requests from multiple tasks, prevents resource overload from causing task failures, and ensures the stable and prioritized execution of core training data tasks.

[0016] In some optional implementations, when multiple task instances need to be deployed, the task instances are sorted in the following order of priority from high to low: the task configuration template used by the task instance to be deployed has a preset number of dedicated cloud resources; the task configuration template used by the task instance to be deployed has a preset number of shared cloud resources; if the dedicated cloud resources and shared cloud resources are already occupied, the task configuration template used by the task instance to be deployed has a higher priority; if the template priorities are the same, the task priority of the task instance to be deployed is higher.

[0017] Based on the aforementioned technical means, tasks with dedicated resources are prioritized for scheduling to ensure that the production of model data with high computing power requirements is not crowded out; tasks with shared resources are secondary, making full use of idle shared computing power; within the same resource type, template and single task priorities are further distinguished. The hierarchical priority mechanism accurately distinguishes the importance of tasks, prioritizing the allocation of resources for key autonomous driving iteration tasks, avoiding low-priority batch tasks from occupying core computing power, and ensuring the progress of model iteration.

[0018] In some optional implementations, the method further includes: obtaining the POD status feedback from each currently running resident POD; determining, based on the POD status, whether the number of resident PODs under any task configuration template is greater than the upper limit of parallel PODs that can be opened in the task configuration template; when the number of resident PODs under any task configuration template is greater than the upper limit of parallel PODs that can be opened in the task configuration template, selecting redundant PODs of the task configuration template to be shut down based on the number of execution nodes in each POD.

[0019] Based on the aforementioned technical methods, the system collects real-time data on the running status of resident PODs, constrains the maximum concurrent number of resident PODs per template, and shuts down redundant PODs with the lowest internal node load when the limit is exceeded. The total number of resident containers is dynamically managed through status feedback to prevent the unlimited creation of resident PODs for similar tasks from exhausting cluster resources. Low-load redundant instances are prioritized for shutdown, while high-utilization resident containers are retained to avoid repeated creation and destruction. Dynamic scaling down of resident containers continuously releases idle computing power to supply other tasks, balancing the concurrent demands of similar tasks with the overall resource capacity of the cluster.

[0020] In some alternative implementations, the stop command is issued when there are no remaining tasks waiting to be processed in the task configuration template corresponding to the target resident POD.

[0021] Based on the aforementioned technical measures, a stop command for the resident POD is only issued when there are no remaining tasks to be processed in the task library, and resources are only released after all tasks within the resident POD have been completed. This avoids data loss and task interruption caused by destroying the container before tasks are completed; resources are released only after all remaining tasks are cleared, ensuring the complete output of batch autonomous driving data. It also prevents frequent starting and stopping of resident containers, extends the reuse cycle of resident PODs, reduces container reconstruction overhead, and ensures the integrity of the data production process and the reliability of results.

[0022] Secondly, the present invention provides a data production apparatus applied to a production platform with schedulable cloud resources. The apparatus includes: a raw perception data acquisition module for acquiring raw perception data; a template acquisition module for acquiring a corresponding task configuration template based on an autonomous driving training task, wherein the task configuration template is a template pre-configured and stored based on the user's data production requirements, and the production requirements correspond to the autonomous driving training tasks; a task generation module for binding the raw perception data and the task configuration template to generate a current task instance; a resource scheduling production module for distributing the current task instance to a corresponding target ordinary POD or target resident POD for execution based on the type of the current task instance, wherein the target ordinary POD is created when the task is distributed and releases cloud resources when the task ends, and the target resident POD is created when the task is first distributed and releases cloud resources after receiving a stop command and all internally running tasks are completed; and a data output module for acquiring training data obtained after the current task instance is executed.

[0023] Thirdly, the present invention provides an electronic device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the method described in the first aspect or any corresponding embodiment thereof.

[0024] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to perform the method described in the first aspect or any corresponding embodiment thereof.

[0025] Fifthly, the present invention provides a computer program product, including computer instructions for causing a computer to perform the method described in the first aspect or any corresponding embodiment thereof. Attached Figure Description

[0026] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0027] Figure 1 This is a schematic diagram of the first process of a data production method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a second process of a data production method according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the database structure according to an embodiment of the present invention; Figure 4 This is a flowchart illustrating the task scheduling process according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the POD operating state according to an embodiment of the present invention; Figure 6 This is a structural block diagram of a data production apparatus according to an embodiment of the present invention; Figure 7 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] It is understood that before using the technical solutions disclosed in the various embodiments of the present invention, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in the present invention and their authorization should be obtained in accordance with relevant laws and regulations through appropriate means.

[0030] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0031] According to an embodiment of the present invention, a data production method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0032] This embodiment provides a data production method that can be used in the aforementioned mobile terminals, such as mobile phones, tablets, etc. (the execution subject is described in conjunction with the actual situation). Figure 1 This is a flowchart of a data production method according to an embodiment of the present invention, the process including the following steps: Step S101: Obtain raw sensor data; Step S102: Obtain the corresponding task configuration template according to the autonomous driving training task. The task configuration template is a template that is pre-configured and stored based on the user's production requirements for data. The production requirements correspond to the autonomous driving training task. Step S103: Bind the original sensing data and the task configuration template to generate the current task instance; Step S104: Based on the type of the current task instance, the current task instance is sent to the corresponding target ordinary POD or target resident POD for execution. The target ordinary POD is used to create when the task is sent and release cloud resources when the task ends. The target resident POD is used to create when the task is sent for the first time and release cloud resources after receiving the stop command and all internally running tasks are completed. Step S105: Obtain the training data obtained after the current task instance is executed.

[0033] Specifically, in the perception data production stage for parallel iteration of multiple models in autonomous driving, existing pipelines struggle to meet the diverse data processing needs of various perception, prediction, and planning models. The indiscriminate reuse and destruction of container resources continuously increases computational overhead. Furthermore, when simultaneously developing and testing multiple models and algorithms, it is often necessary to produce multiple, differently distributed, and scenario-specific versions of training data. Each type of data requires a complex and potentially varied data processing pipeline. In summary, addressing the diverse and multi-version training data production needs arising from parallel development of multiple models and versions requires building a data production system that is fast-responding, flexibly scheduled, and resource-efficient.

[0034] Based on the hierarchical data processing and container resource classification approach, this invention provides a standardized method for producing perception data. The following section will provide a concrete explanation using a training scenario for autonomous driving on urban roads for passenger vehicles.

[0035] The vehicle manufacturer's intelligent driving data center collects a large amount of raw vehicle perception data every day. The data sources cover, but are not limited to, vehicle perception devices such as front-view cameras, LiDAR, millimeter-wave radar, and vehicle IMU. A single data packet contains continuous time-series images, point clouds, and coordinate positioning information. Such raw perception data that has not been standardized cannot be directly used for model training. The first step in the entire production chain is to complete the unified collection and structured reading of all vehicle perception data sources.

[0036] The production platform provided in this embodiment of the invention, such as Figure 2 As shown, it connects to the original data source, data filtering warehouse, data processing warehouse, computing cluster, cache (PFS), runtime record database (DB), and storage cluster.

[0037] The production platform responds to manual task issuance and management instructions from R&D personnel, or to instructions that are automatically triggered according to a certain period or event, to begin binding the raw perception data and task configuration templates, and then proceeds with subsequent training data production tasks.

[0038] The original data source is the location used to store the aforementioned large amount of vehicle-mounted raw perception data. The distributed file system performs bucket caching on the data packets returned by each test vehicle. The program polls the unprocessed data packets in the cache directory, and after completing the data packet header verification and time sequence index parsing, it extracts the raw perception data entities that can participate in subsequent processing.

[0039] After fully extracting the compliant raw perception data, the system needs to anchor the model development requirements corresponding to this data production, and retrieve the pre-saved standardized task configuration template to uniformly constrain the execution standards of the entire data processing chain.

[0040] Internally, the company pre-configures and archives multiple templates based on the training objectives of different R&D branches. Each template is bound to a specific training task, with one template corresponding to a type of heterogeneous training task. For example, perception model iteration tasks, path prediction model iteration tasks, and vehicle motion planning model iteration tasks belong to three independent autonomous driving training tasks, each corresponding to a completely independent task configuration template. After R&D personnel select the target training task identifier on the front-end interface, the back-end retrieval service matches and loads the template file bound to that task identifier from the database. The template internally establishes the basic specifications for the entire process of processing this type of training data, eliminating the need for staff to repeatedly enter processing parameters and avoiding the problem of inconsistent processing standards across multiple batches of data from the source. The task configuration template refers to a standardized set of processing configurations that staff pre-edit and persistently store in the operation record database based on the data production needs of a specific model training task, serving as a unified execution benchmark for raw data processing.

[0041] After retrieving the template file, the production platform associates and binds the read raw perception data entities with the matched template configuration information to generate a task instance with complete execution attributes. This task instance is then distributed to the computing nodes of the computing cluster as a cluster scheduling unit. The computing cluster is a cluster of devices in a distributed cloud cluster used to execute computing tasks.

[0042] For single, small-scale sensing data packets, the system directly completes the association mapping between data fields and template parameters. For massive data packets transmitted back in batches by a fleet, this embodiment of the invention divides the batch data into data units according to single data units. Each independent data unit is bound to the same set of template configurations, and each generates an independent task instance. The task instance is the smallest scheduling unit generated after the original sensing data and template configuration are bound together, carrying all the basic information required for the entire process of a single data processing.

[0043] As production demands accumulate, the scheduling database gradually accumulates instances of tasks to be executed. Meanwhile, the production platform distributes these instances to two types of isolated container resources for computation based on the task type identifier carried by each instance. The lifecycle management rules for the two types of containers are clearly distinguished.

[0044] The standard POD container is created when a task instance requests computing power. Once all data processing steps within the instance are completed, the cloud resources such as CPU, GPU, and memory occupied by the container are immediately released. The resident POD container is only initialized when a similar task instance is first issued. It is not destroyed after a single instance finishes execution; it is only reclaimed after responding to a termination command issued by the platform and after all queued instances within the container have completed processing. For example, if six task instances all belong to temporary road scene sampling data processing, the system will create a separate standard POD for each instance, with the six containers operating synchronously in parallel. Once a single instance completes processing, the corresponding container is destroyed. If it is a continuous 7-day task producing data on obstacles in a fixed urban area, the platform will initialize a resident POD, continuously issuing similar task instances to this container over 7 days until all data processing is completed. Then, a termination command is issued from the background, the container clears its internal queued tasks, and releases its computing power. The POD is the smallest independent computing power scheduling carrier in a cloud-native cluster, encapsulating a complete data processing runtime environment. Cloud resources refer to reusable computing and storage resources such as CPU cores, graphics computing units, running memory, and local disks that are uniformly allocated in a computing cluster.

[0045] After receiving a task instance, the container needs to perform preprocessing operations such as data filtering, processing, cleaning, and alignment before executing the formal data transformation according to the training task to ensure data quality. The relevant code for data preprocessing operations is extracted from the data filtering repository and the data processing repository.

[0046] The process involves first filtering the raw data, followed by data processing. When multiple POD containers need to perform the same data preprocessing, the first POD container to process the data retrieves the relevant code from the data filtering and processing repositories, executes it, and places the code in a cache (PFS). This allows subsequent POD containers to quickly retrieve the code from the cache, accelerating the task execution process.

[0047] During task execution, relevant log files are stored in the runtime database (DB), such as... Figure 3 As shown, it includes task information, template information, task execution information, and resource information about cloud resources.

[0048] Finally, after each POD container completes its corresponding task instance, the system automatically extracts the standardized structured dataset and outputs it to the persistent storage cluster as training data that can be directly used for autonomous driving model iteration.

[0049] This invention starts with the acquisition of raw vehicle-mounted sensor data, proceeds through template matching, task instance generation, and differentiated container scheduling, and finally produces a standardized training dataset. Compared to traditional cloud resource scheduling models, this invention uses task templates to solidify processing standards, eliminating batch data discrepancies caused by repetitive manual configuration. Furthermore, through differentiated lifecycle management of ordinary and resident containers, resident containers can be reused in high-frequency, similar data processing scenarios, eliminating the computational overhead of repeatedly creating and destroying containers, such as pulling operating system images and initializing the environment. This significantly reduces overall computational resource consumption in scenarios involving large-scale daily processing of sensor data. Simultaneously, the fully automated workflow eliminates the need for manual intervention in data transfer and parameter configuration, continuously meeting the massive, high-frequency data output demands of synchronous iterations of multi-branch autonomous driving models.

[0050] In some optional implementations, the steps of configuring the task configuration template in this embodiment of the invention include: Step a1: Configure data processing version information, resource and environment configuration information, processing link parameters, and task management identifier. Data processing version information is used to specify the operator version used for data filtering and processing. Resource and environment configuration information is used to define the hardware resource information for task execution. Processing link parameters are used to represent the execution parameters required by the data processing operators. Task management identifier is used to distinguish task types and mark task priorities.

[0051] Specifically, in the parallel development phase of multi-branch intelligent driving, different iterations of perception and prediction algorithms require different data processing operators and hardware computing power. Each data production task involves repeatedly entering the entire set of operating parameters, which causes a lot of redundant manual operations. Parameter entry deviations can also lead to the fragmentation of processing standards for multiple batches of training data. This embodiment of the invention explains the standardized construction process of task configuration templates.

[0052] After logging into the intelligent driving data production platform, R&D personnel can access the template creation interface to build a data processing execution blueprint suitable for their own business branch from scratch. The entire template retains processing rules and operational constraints, but is not bound to any batch of raw vehicle perception data to be processed. After configuration, it will be persistently written to the platform's backend distributed relational database, forming a standardized configuration file that can be repeatedly retrieved and reused. The template building process is divided into four configuration dimensions.

[0053] The first type of configuration item is data processing version information. This field is used to lock the fixed iteration version of the operators called throughout the entire data cleaning and feature processing chain, and at the same time define the version baseline of the data filtering logic. Operators refer to the program operation units that perform filtering, enhancement, and ground truth annotation on point cloud, image, and millimeter-wave sensor data. Different versions of operators will output datasets with different feature distributions.

[0054] After completing the operator version baseline setting, the next step is to input parameters for resource and environment configuration. This module is used to select the computing power resource pool and precisely define the hardware resource thresholds allocated during task execution. A resource pool refers to a collection of computing power nodes within a cloud cluster, isolated according to business lines. Hardware resources include physical computing power indicators such as the number of CPU cores, RAM capacity, graphics accelerator card model and memory quota, and local temporary disk space. The resource pool selection strategy is defined according to user needs. For example, it includes cloud resources from multiple vendors such as Alibaba Cloud and Tencent Cloud, allowing users to specify a particular cloud vendor's resource pool. Options include automatic allocation and multiple selection based on the selected strategy. Furthermore, tasks can be assigned to a specific machine based on user needs, addressing the issue of special tasks requiring large amounts of resources while other resources are scattered and insufficient to form a large resource pool. During task instance assignment and scheduling, the production platform directly reads the hardware parameters from the template to complete computing power matching, eliminating the need for manual secondary verification of resource specifications.

[0055] After configuring and customizing the link parameters, these parameters become the dedicated control variables required for the operation of various operators within the entire data processing pipeline. They represent the refined constraints for operator execution. Multimodal alignment operators, obstacle threshold filtering operators, and image enhancement operators all have independently adjustable parameters. For example, the point cloud and image temporal alignment operator has a time offset compensation threshold, the obstacle filtering operator has a minimum pixel size threshold, and the image enhancement operator has a brightness adjustment range.

[0056] Finally, the task management identifier is defined. This set of fields serves a dual purpose: dividing business logic and classifying task priorities, providing a basis for determining the resource allocation order in subsequent cluster scheduling. The identifier content includes two core categories: business line codes and task priority tags. Business line codes distinguish between R&D branches such as perception, prediction, and planning. Priorities are divided into multiple tiers; for example, priority P0 corresponds to urgent tasks for algorithm model iteration, P4 corresponds to regular batch data archiving tasks, etc. This is just an example and is not a limitation. This identifier is stored permanently with the template. After the scheduling service reads the template identifier bound to the task instance, it can accurately distinguish the task's business affiliation and urgency, providing a benchmark for the computing power allocation order.

[0057] After all template configurations are filled in, submit the save command. The platform will write the integrated template configuration information into a separate data table in the database. The data table stores rule-type parameters but does not record any original vehicle-mounted perception data packets. When initiating data production tasks later, developers only need to retrieve the template number that matches their business from the template library to select the appropriate template. There is no need to repeatedly fill in complex configurations such as operator version, hardware specifications, link parameters, and business identifiers, significantly reducing the manual operation time required to start the task.

[0058] This invention comprehensively covers the operator logic, hardware support, fine-grained computation, and scheduling hierarchical constraints of data processing through four configuration dimensions, and uniformly solidifies them into template files. This eliminates data standard deviation caused by manual repetitive parameter entry, and maintains a high degree of consistency in the feature distribution of multiple batches of training datasets. The complete configuration is persistently stored and supports unlimited reuse, significantly reducing the manual operation costs in large-scale data production scenarios.

[0059] In some optional implementations, step S103 above includes: Step b1: If the original sensing data is a single data point, then the original sensing data and the task configuration template are directly bound together to generate the current task instance; Step b2: If the original sensing data is batch data, then the original sensing data is divided into multiple independent data units; Step b3: Bind each data unit to the task configuration template to generate multiple corresponding current task instances.

[0060] Specifically, after R&D personnel select a pre-stored task configuration template, they submit the vehicle sensor data to be processed on the platform's data entry interface. The platform's backend will first classify and determine the storage volume and file organization of the submitted data, distinguishing between single independent sensing data packets and batch aggregated datasets. Then, it will perform the association and binding operation between the data and the template configuration according to the path, and finally produce task instances that can be scheduled by the cluster. All instance records are uniformly and persistently stored in the platform's task scheduling database, waiting for the scheduling service to pull and distribute them to the computing container.

[0061] When the backend determines that the submitted object is a single piece of raw sensing data, the system no longer performs data fragmentation and segmentation operations. Instead, it directly completes the field association mapping between the sensing data entity and the loaded template configuration information, generating a complete single task instance in one go. A single piece of raw sensing data refers to an independent time-series data packet generated during a single vehicle-mounted road test trip, used for small-scale control experiments by the algorithm team. The platform reads the metadata of this data packet, such as the object storage path, timestamp, and sensor calibration parameters, and couples it with the configuration fields such as operator version, hardware resources, link parameters, and scheduling identifier fixed in the template to generate a uniquely identified task instance. The instance also carries the data storage address and a complete set of processing constraint parameters, fully recording all the preconditions for this data processing, and is directly written to the task database to await scheduling.

[0062] If the backend identifies the user-submitted content as a batch-aggregated sensor dataset, it first initiates data sharding and splitting logic to break down the complete batch data into multiple independent data units with non-overlapping boundaries and balanced sizes. Batch data refers to the aggregated sensor data packets transmitted back by test vehicles. The platform's built-in sharding rules use a preset volume as the single unit size threshold, automatically traversing all sub-files within the batch dataset, and cutting and grouping them according to the continuity of the acquisition time sequence. Finally, multiple independent data units are obtained, each containing a complete and continuous vehicle sensor time sequence, without data fragmentation issues across trips or time periods. After splitting, the system assigns a unique unit number to each data unit for subsequent instance tracing.

[0063] After completing the batch data sharding to obtain multiple independent data units, the platform traverses all sharded units and binds and couples the sensor data metadata in each unit with the same set of task configuration templates. The combination relationship between each set of data units and the template corresponds to the generation of an independent task instance.

[0064] Based on the above technical methods, small-volume data sets eliminate the need for sharding and splitting processes, reducing CPU consumption for background file traversal and data segmentation, and lowering the latency of task instance generation for small-scale experimental samples. Large-volume datasets are automatically and evenly sharded, avoiding the problem of a single ultra-large dataset monopolizing a single computing container. The computing power of multiple nodes in the cluster can be simultaneously distributed among multiple sharded tasks, improving the overall data processing throughput.

[0065] In some optional implementations, step S104 above includes: Step c1: If the current task instance is defined as a normal task, then when the idle cloud resources meet the availability conditions of the current task instance, create a target normal POD for the current task instance. Step c2: Deploy the current task instance to the target ordinary POD for execution, and release the target ordinary POD when the current task instance finishes execution; Step c3: If the current task instance is defined as a resident task, determine whether there is a first task instance that was issued and executed before the current task instance. The first task instance and the current task instance have the same attribute information. If step c4 does not exist, then when the idle cloud resources meet the availability conditions of the current task instance, create a first target resident POD for the current task instance and send the current task instance to the first target resident POD for execution. Step c5: If it exists, find the second target resident POD that is running the first task instance, and when the second target resident POD has the resource conditions to receive the current task instance, send the current task instance to the second target resident POD for execution.

[0066] Specifically, in response to the two different types of task requirements that exist simultaneously in the autonomous driving data production scenario, the platform backend deploys two independent scheduling engines in parallel to undertake the computing power allocation work for different types of tasks. The two engines periodically poll the task database and cluster resource monitoring panel to complete the resource balance inventory, task selection and distribution operations in real time.

[0067] Scheduler A manages all resources during the initial creation of both regular and persistent tasks. Scheduler B is dedicated to maintaining the continuous replenishment and status monitoring of persistent container instances. All POD instances within the cluster continuously push running metrics to the platform's memory status manager, providing real-time data support for various decision-making operations in the scheduling branch.

[0068] The POD in this embodiment of the invention is the smallest independent operating unit of a cloud-native computing cluster, internally encapsulating a complete data processing and running image and operator dependency environment. Task attribute information includes four core indicators: task configuration template number, operator code version, data processing script, and task batch. Two task instances are considered to have the same attribute if at least the task configuration template number matches completely. The more matching indicators, the higher the degree of attribute matching between the tasks. If even the task configuration template number does not match, the two tasks are considered to have different attributes.

[0069] If a task is defined as a regular task by the user, scheduler A needs to participate. Scheduler A first traverses all computing power nodes in the cluster, calculates the total CPU, GPU memory, and memory already occupied by various running instances, and combines this with the cluster's total resource threshold to calculate the current idle computing power scale. This determines whether the idle cloud resources can cover the predefined hardware quota of the task instance template. If the idle computing power is sufficient, the scheduling service issues a creation command to the cluster nodes, initializes the target regular POD, and completely deploys the task instance into the container to start computation. After all data cleaning, alignment, and labeling processes within the container are completed, and the platform detects the instance execution completion flag, it immediately releases all cloud resources occupied by the target regular POD, destroys the container runtime environment, and frees up hardware computing power for other pending task instances.

[0070] If the scheduling engine identifies that the instance to be deployed belongs to the resident task branch, it first checks whether there is a historical resident task whose first task instance with completely matching attributes has been executed or is being executed, and the resident POD executing the first task instance has not been destroyed.

[0071] Scheduler B executes multi-stage matching and judgment logic. Its core objective is to reuse existing resident containers of the same specification, avoiding the environment initialization overhead caused by repeatedly creating containers. Scheduler B retrieves all running resident instance data from the real-time cache of the memory state manager, searches for the first task instance that was previously issued and is still running, and verifies whether the attribute information of this historical instance is consistent with that of the current instance to be issued, i.e., at least the template number matches.

[0072] If the memory state manager finds a first task instance with matching attributes, the system simultaneously locks the second target resident POD that will run that historical instance. It then reads the real-time status data reported by the container, extracting indicators such as the remaining capacity of the internal task queue, current video memory and memory usage, etc., to determine whether the resident container has sufficient remaining resource space to accept new task instances. The platform presets a water level threshold for each type of resident POD. For example, the maximum number of queued tasks in a single resident POD is 20. If only 6 tasks remain in the queue inside the container, below the preset water level limit, it means the container has the resource conditions to receive new tasks. Scheduler B directly retrieves the currently pending instance from the task database and pushes it to the internal task queue of the second target resident POD for execution. If the queued tasks inside the container are full and the hardware usage is close to the threshold, the task delivery is temporarily suspended, waiting for the next round of status feedback updates to re-verify the container load until the container frees up enough space to complete the task replenishment.

[0073] If the memory search results do not show any historical resident task instances with matching attributes, it means that there are currently no reusable running containers within the cluster. In this case, the system will switch back to scheduler A. Scheduler A, in conjunction with the resource statistics module, will calculate the current idle computing power of the cluster and verify whether the remaining hardware resources meet the hardware operation standards defined by the resident task template. If the cluster's idle computing power meets the standards, a creation command will be sent to the compute nodes to generate the first target resident POD. The current task instance will be pushed into the container to start computation. This container instance will remain resident in the cluster and will not be destroyed when a single instance's computation ends.

[0074] During the resident container's runtime phase, it continuously reports its real-time running status to the platform. This reporting includes the unique ID of the currently processed task, the computing node number, hardware load, and the amount of tasks in the internal task queue. All status data is written to the memory status manager in real time. Scheduler B's operations are based on this updated status data, ensuring timely decisions regarding resource verification, container reuse, and task replenishment. The platform only issues a termination command to the resident POD when there are no pending tasks in the corresponding template. After all queued instances within the container have completed their computations, the cloud resources occupied by the resident container are reclaimed.

[0075] The dual-engine scheduling strategy provided in this invention employs a container lifecycle management strategy where ordinary tasks are created on demand and destroyed immediately upon completion. Temporary discrete tasks do not occupy cluster computing power for extended periods, ensuring that limited hardware resources can be cyclically supplied to meet multiple batches of temporary data processing needs. Permanent tasks with the same attributes prioritize reusing existing running containers, avoiding network and computing power losses caused by repeatedly pulling images and initializing operator environments. Based on the real-time status feedback mechanism of permanent containers, the scheduling engine can accurately grasp the load margin within containers and dynamically replenish tasks to low-load permanent containers, fully utilizing idle computing power within permanent containers and increasing the data processing throughput limit of a single permanent POD. The dual scheduling engines operate independently, with resource statistics and distribution management processes for ordinary and permanent tasks not interfering with each other. Parallel scheduling of the two types of tasks avoids resource contention and task blocking issues, significantly improving the overall data output throughput of the cluster in large-scale intelligent driving data concurrent production scenarios.

[0076] In some alternative implementations, the steps of performing internal tasks via a target ordinary POD or a target resident POD include: Step d1: Divide all task instances within the current POD into multiple execution nodes according to the smallest execution unit; Step d2: Select multiple execution nodes that do not conflict with each other to execute concurrently within the current POD, according to the execution order of each execution node and the type of hardware resources used.

[0077] Specifically, the multimodal data processing chain for autonomous driving includes multiple sequentially dependent computational steps and multiple hardware resource monopolies. In traditional processing modes, the entire task is executed sequentially. When GPU computing power is idle, the CPU waits continuously; when the CPU is fully loaded, graphics acceleration resources are idle. The utilization rate of single container hardware resources remains low for a long time, and the problem of wasted container computing power is particularly prominent in scenarios of large-scale perception data production. This invention provides a layered decomposition and conflict determination parallel execution scheme for the internal task computation process of POD.

[0078] After the ordinary POD and resident POD carrying out data processing start the task processing flow, they do not directly and serially execute the entire chain of operators for a single task instance. Instead, they first perform hierarchical decomposition processing on all task instances to be processed within the container, breaking them down level by level to the smallest execution node that cannot be further subdivided. The smallest execution unit refers to a single, independent atomic operator operation step with clear resource consumption characteristics, and there are no sub-operation logics that can be further divided. An execution node is an independent process unit that carries the atomic operation step, and each node is bound to only one type of fixed hardware resource consumption requirement. Taking a single urban perception task instance as an example, the complete processing chain includes five core processes: raw point cloud denoising, image distortion correction, multimodal temporal alignment, obstacle feature extraction, and ground truth label mapping. The platform will further decompose each process downwards. The point cloud denoising process is broken down into spatial filtering nodes and outlier removal nodes; the image correction process is broken down into intrinsic parameter calibration nodes and pixel mapping nodes; the temporal alignment process is broken down into timestamp matching nodes and frame synchronization interpolation nodes; the feature extraction process is broken down into GPU feature encoding nodes and dimensionality compression nodes; and the label mapping process is broken down into coordinate transformation nodes and label file generation nodes. The entire task instance is ultimately decomposed into 10 independent execution nodes, all of which are uniformly included in the POD's internal process scheduling pool awaiting computing power allocation. Assuming that 5 task instances of the same specifications are cached simultaneously within the container, the system will sequentially split the 5 instances, generating a total of 50 independent execution nodes. The container's built-in resource management program uniformly manages the scheduling sequence of all nodes.

[0079] After all task nodes are broken down and aggregated, the container's built-in resource management program synchronously reads two core attributes of each execution node. The first is the pre-defined execution order constraints for each node, marking whether there are strong data dependencies between nodes; the second is the type of hardware resources required for node operation, distinguishing between pure CPU computing nodes, GPU graphics computing nodes, and memory-intensive nodes. The program performs conflict determination based on these two sets of attributes. For example, exclusive nodes of the same hardware resource type cannot run in parallel simultaneously; nodes with prerequisite output dependencies cannot start before the preceding node completes. Finally, multiple groups of nodes without resource preemption conflicts or data dependency conflicts are selected and started concurrently for computation.

[0080] This invention, based on the container's built-in process monitoring module, collects hardware usage metrics of each node in real time and dynamically adjusts the combination of concurrent nodes. It avoids forcibly scheduling nodes with dependencies or resource conflicts in pursuit of resource saturation, thus preventing data processing errors and operator operation crashes from the underlying level.

[0081] This invention breaks down tasks layer by layer to the smallest atomic execution node, avoiding the inherent constraint of sequential execution of the entire task and unleashing the potential of container multi-hardware parallel computing. Conflict verification is performed through a dual dimension of resource type and data dependency, scheduling only conflict-free nodes to run synchronously. This ensures that while maximizing hardware resources, the accuracy of data processing results is maintained, reducing the probability of operator errors and data feature corruption. Finally, the container's built-in process-level monitoring program tracks the load status of each node in real time and dynamically adjusts the combination of concurrent nodes, achieving continuous saturation of computing resources within the POD. With the same cluster hardware scale, it can support larger-scale autonomous driving perception data production tasks, reducing the hardware investment costs associated with expanding the computing power cluster.

[0082] In some optional implementations, steps c1 and c4 above include: Step e1: Determine the number of tasks that can be issued in this cycle based on the current idle cloud resources; Step e2: Sort the task instances based on the priority of the current task instance and the priority of other task instances within the current period, and determine whether the current task instance is within the number of tasks. Step e3: If the current task instance is within the task limit, create a target ordinary POD or a target resident POD for the current task instance and update the current idle cloud resources.

[0083] Specifically, such as Figure 4 As shown, the computing power supply of the intelligent driving data production cluster has a fixed upper limit. Multiple batches of task instances with different urgency levels will continuously flood into the scheduling queue. If the total amount of resources is not calculated and priority is not sorted, it is very easy for high-value model iteration tasks to be squeezed out by low-priority archiving tasks, and for the cluster resources to be overloaded and crash due to excessive task issuance.

[0084] In this embodiment of the invention, scheduler A periodically initiates a resource inventory process according to a fixed time slice. It performs resource utilization statistics on all computing power nodes across the cluster, summarizing the CPU cores, graphics accelerator cards, RAM, temporary storage, and other hardware metrics consumed by all running ordinary PODs and resident PODs. The total cluster resource quota is subtracted from the occupied resource value to obtain the total amount of idle cloud resources available for the current scheduling cycle. The scheduler then discounts the idle resources against the hardware requirements preset in a single task instance template to calculate the maximum number of tasks that can be safely deployed in the current scheduling cycle without triggering resource overload.

[0085] Taking the standard computing cluster in the factory area as an example, the total cluster quota is 128 CPU cores, 8 A100 40GB graphics cards, and 512GB of RAM. Currently, the containers running in the cluster occupy a total of 64 CPU cores, 3 graphics cards, and 220GB of RAM, leaving 64 CPU cores, 5 graphics cards, and 292GB of RAM as idle resources. The tasks waiting to be dispatched in the queue are all urban sensing processing tasks. Each instance has a fixed requirement of 4 CPU cores, 0.5 A100 graphics cards, and 16GB of RAM. The scheduler calculates through resource matching that a maximum of 12 task instances can be dispatched in parallel in this round, which is the upper limit of the task quota that can be dispatched in this round.

[0086] After calculating the total number of tasks that can be issued in a single cycle, the scheduler reads all instances in the task database that are in a pending scheduling state, extracts the predefined priority identifier from the binding template of each instance, and reorders all pending scheduling instances in descending order based on their priority. The lower the priority value, the higher the urgency of the business. P0 is for urgent processing tasks related to model iteration, while P1 to P4 correspond to low-urgency tasks such as regular batch processing, data archiving, and sample backup, respectively.

[0087] The scheduler selects the number of task instances at the top of the sorting queue as the target tasks approved for issuance in this round, while the remaining low-priority instances are stored in the task database and carried over to the next scheduling cycle to participate in resource verification and sorting again.

[0088] After determining that the current pending task instance falls within the quota interception range of this round, the scheduler sends a container initialization instruction to the cluster nodes. Based on the task type marked by the instance, it creates either a target ordinary POD or a first target resident POD. After the container is created and the runtime environment is verified, it synchronously updates the global idle resource ledger of the cluster, deducts the hardware resource value occupied by the current instance, and corrects the idle resource base that can participate in the calculation in the next scheduling cycle.

[0089] This invention provides a complete closed-loop scheduling process encompassing resource quota calculation, priority ranking, and resource ledger updates. Each round of scheduling is based on real-time idle computing power calculation and upper limit allocation, preventing uncontrolled task allocation from causing cluster resource exhaustion and task process freezes, thus ensuring the long-term stable operation of the computing cluster. Business priority is used as the core criterion for task selection, prioritizing high-value tasks such as model iteration to occupy computing resources without being hindered by low-priority data archiving tasks, matching the business demands of rapid iteration in autonomous driving algorithms. The cluster resource ledger is updated synchronously after container creation, and the resource calculation benchmark for each round of scheduling is a real-time, accurate value, significantly reducing the calculation error of resource allocation and significantly improving the accuracy of computing resource allocation.

[0090] In some optional implementations, when multiple task instances need to be deployed, they are sorted in descending order of priority as follows: The task configuration template used by the task instance to be deployed has a preset number of dedicated cloud resources; the task configuration template used by the task instance to be deployed has a preset number of shared cloud resources; when dedicated cloud resources and shared cloud resources are already occupied, the task configuration template used by the task instance to be deployed has higher priority; when the template priorities are the same, the task priority of the task instance to be deployed is higher.

[0091] Specifically, in the autonomous driving R&D system, different business branches simultaneously submit massive data production tasks to the computing power cluster. Some core algorithm iteration tasks are configured with dedicated computing power hardware, while regular tasks can call the cluster's shared computing power pool. Task sorting is completed through a single-dimensional priority, which easily leads to imbalances such as low-value tasks occupying dedicated computing power scheduling channels and high-importance template tasks being delayed. This embodiment of the invention constructs a four-layer progressive sorting judgment rule, which uses resource ownership type, template preset level, and the priority of the individual task as weights to complete the queue reordering.

[0092] After scheduler A calculates the total number of tasks that can be dispatched in each cycle, it will batch read all the scheduled instances in the task database and extract the three core identifiers embedded in the template of each instance: template resource allocation type, template global priority tag, and instance's own task priority tag. Then, it will complete multiple rounds of filtering and sorting according to a fixed hierarchical order. The four judgment dimensions have absolute priority from top to bottom. When there is differentiation in the upper-level dimension, the sorting is completed directly according to the upper-level rules. Only when all instance attributes in the upper-level dimension are completely consistent will they enter the next level for secondary differentiation.

[0093] The first ranking criterion is whether the template is configured with a pre-defined scale of dedicated cloud resources; this level has the highest ranking weight. Dedicated cloud resources refer to dedicated computing hardware that is pre-allocated within the cluster for specific business templates and is not shared externally. For example, the emergency iteration template for the perception model is equipped with a dedicated computing power partition consisting of two A100 graphics cards and a 32-core CPU. Shared cloud resources refer to a public computing power pool that is open across the entire cluster and can be accessed by multiple templates. The scheduler will place all task instances bound to templates with dedicated resources at the front of the queue, and the remaining shared resource template instances at the back.

[0094] After completing the resource type stratification, the second-level judgment dimension is entered. If there are still idle cloud resources after the tasks are issued in the first level, the judgment is based on the global priority preset of the template. The enterprise uniformly classifies all templates into five priority levels from P0 to P4. Remaining tasks that do not belong to the first-level resource type and have a higher template priority are issued first, so as to prioritize the issuance of high-weight template tasks except for designated exclusive and shared resources.

[0095] When two or more task instances simultaneously match the first two judgment criteria—that is, the resource ownership type is consistent and the global priority of the bound template is completely equal—then the third judgment dimension is activated, comparing the task priority tag carried by each individual task instance. Task priority is the urgency level independently marked by the developer when initiating the task, distinguishing single business needs beyond the template's unified weight. For example, under the same P0-level perception template, there might be a P0 urgent sample task for offline algorithm optimization and a P2 regular task for daily incremental labeling. The scheduler prioritizes instances with higher task priority, allocating computing power containers first to urgent optimization samples, and then scheduling regular incremental labeling tasks accordingly.

[0096] This invention places dedicated resource template tasks with the highest priority in scheduling, and prioritizes the allocation of dedicated computing power partitions to core model iteration services. This ensures that the progress of autonomous driving algorithm iteration is not interfered with by regular data archiving tasks from the scheduling level. Resource types, template levels, and single task urgency are differentiated layer by layer, and multi-dimensional weighting avoids the scheduling bias caused by a single priority label. The task issuance order is highly matched with the actual importance of the business. When disputes arise in computing power allocation during batch data production, the sorting criteria can be traced back layer by layer, making it easy for operations and maintenance personnel to quickly locate the root cause of the scheduling order deviation.

[0097] In some optional embodiments, the data production method provided by the present invention further includes: Step f1: Obtain the POD status reported by each currently running resident POD; Step f2: Determine whether the number of resident PODs under any task configuration template is greater than the maximum number of PODs that can be opened in parallel for that task configuration template based on the POD status. Step f3: When the number of resident PODs under any task configuration template exceeds the limit of parallel PODs that task configuration template can open, select the redundant PODs of the task configuration template to close based on the number of execution nodes in each POD.

[0098] Specifically, when multiple resident PODs are running in parallel under a pre-existing template, tasks from other templates may also need to be executed. In this case, the pre-existing template needs to shut down some PODs to free up available cloud resources for other templates. Users can define a limit on the number of PODs that can be run in parallel for templates that need to shut down, based on actual needs. This limit restricts the maximum number of resident containers that a single type of service can simultaneously reside in the cluster, preventing a single service from consuming computing power indefinitely.

[0099] All resident PODs running within the cluster maintain a long-lived connection to the production platform backend, continuously pushing their full-dimensional running status data to the platform's memory state manager. Scheduler B periodically pulls all resident POD status information cached in memory, completing the unified collection of all resident instance data. This process corresponds to the status collection phase of this method. The POD status includes the specific task ID currently being processed, the task running node name, and the current running status, where the current running status is as follows: Figure 5 As shown, it includes initialization, pause, running, and termination.

[0100] After scheduler B completes the collection of the status of all resident PODs, it will group all resident instances according to the template number and compare the real-time total number of resident PODs in each group with the upper limit threshold of parallel PODs pre-entered for that template to determine whether the scale of resident containers of a single template exceeds the preset control boundary.

[0101] When the total number of resident PODs for a certain task configuration template exceeds the preset parallel limit, the scheduler retrieves the status data reported by all resident PODs under that template group, extracts the total number of existing execution nodes inside each container as the load judgment criterion, and selects the top preset number of containers with the fewest execution nodes as redundant PODs. A termination command is then issued to complete the container shutdown and resource reclamation. The number of execution nodes directly represents the current load level of the container; the fewer the number of nodes, the higher the idle level of the container's computing power. Prioritizing the shutdown of low-load containers can maximize the preservation of high-load, high-output resident computing power carriers, avoiding the interruption of batch tasks caused by shutting down high-load containers.

[0102] Based on the aforementioned technical measures, and utilizing the real-time long-connection status reporting mechanism of resident PODs, runtime metrics are cached in memory in real time, eliminating the need for frequent disk database queries and ensuring real-time response capabilities for management actions. Furthermore, a separate parallel limit for resident containers is set for each type of business template, limiting the long-term monopolization of cluster computing power by a single business from the source. This balances the computing power allocation among multiple branches of intelligent driving services, including perception, prediction, and planning, avoiding the imbalance problem of a single business crowding out all shared computing power. In addition, redundant container screening is based on the existing number of internal execution nodes, prioritizing the shutdown of low-load idle containers while maximizing the retention of high-throughput computing power carriers. This reduces the total size of resident containers while minimizing the reduction in overall data processing capacity of the template. Finally, resources are reclaimed only after all execution nodes within a container have completed their computations, preventing forced interruptions to the ongoing data processing flow and eliminating production failures such as missing perception datasets and mislabeled labels caused by forced container destruction, thus ensuring the integrity and accuracy of autonomous driving training data output.

[0103] In some alternative implementations, the stop command is issued when there are no remaining tasks waiting to be processed in the task configuration template corresponding to the target resident POD.

[0104] Specifically, if a resident container is released directly while there are still tasks pending in the template, it will cause unprocessed perception data to remain in the queue and disrupt the data production chain. This embodiment of the invention limits the pre-judgment conditions for issuing termination commands. Scheduler B periodically searches the task database to count the total number of unstarted, queued task instances under the template to which the target resident POD belongs. Only when the search result is zero and no tasks awaiting processing remain will a stop command be issued to that resident POD. The scheduler can only release cloud resources when all batch perception data shards have been deployed, all queued nodes within the container have completed their calculations, and there are no remaining tasks pending in the template. Through these technical means, the integrity of the entire data production process is ensured, reducing the overhead of image pulling and environment initialization caused by repeatedly rebuilding resident containers, and ensuring stable full-volume production of autonomous driving training data.

[0105] This embodiment also provides a data production apparatus for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0106] This embodiment provides a data production device, such as... Figure 6 As shown, it includes: Raw sensing data acquisition module 601 is used to acquire raw sensing data; The template acquisition module 602 is used to acquire the corresponding task configuration template according to the autonomous driving training task. The task configuration template is a template that is pre-configured and stored based on the user's production requirements for data. The production requirements correspond to the autonomous driving training task. The task generation module 603 is used to bind the raw sensing data and the task configuration template to generate the current task instance; The resource scheduling production module 604 is used to distribute the current task instance to the corresponding target ordinary POD or target resident POD for execution based on the type of the current task instance. The target ordinary POD is created when the task is distributed and releases cloud resources when the task ends. The target resident POD is created when the task is distributed for the first time and releases cloud resources after receiving the stop command and all internally running tasks are completed. The data output module 605 is used to obtain the training data obtained after the current task instance is completed.

[0107] In some alternative implementations, the task generation module includes: The first binding unit is used to directly bind the original sensing data and the task configuration template to generate the current task instance if the original sensing data is a single data item. The splitting unit is used to divide the original sensing data into multiple independent data units if the original sensing data is in batches. The second binding unit is used to bind each data unit to the task configuration template respectively, generating multiple corresponding current task instances.

[0108] In some alternative implementations, the resource scheduling production module includes: The ordinary POD creation unit is used to create a target ordinary POD for the current task instance if the current task instance is defined as an ordinary task and the idle cloud resources meet the availability conditions of the current task instance; The normal task execution unit is used to send the current task instance to the target normal POD for execution, and release the target normal POD when the current task instance finishes execution; The persistent task analysis unit is used to determine, if the current task instance is defined as a persistent task, whether there is a first task instance that was issued and executed before the current task instance, and the first task instance has the same attribute information as the current task instance. The resident POD creation unit is used to create a first target resident POD for the current task instance if it does not exist, and to distribute the current task instance to the first target resident POD for execution when the idle cloud resources meet the availability conditions of the current task instance. The resident POD calling unit is used to find the second target resident POD that is running the first task instance if it exists, and to send the current task instance to the second target resident POD for execution when the second target resident POD has the resource conditions to receive the current task instance.

[0109] The apparatus provided in this embodiment of the invention can execute the method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects for executing the method. Further functional descriptions of the various modules and units described above are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0110] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0111] The following is a detailed reference. Figure 7This diagram illustrates a suitable structural schematic for implementing an electronic device according to embodiments of the present invention. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 701, which can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) 702 or a program loaded from memory 708 into random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the electronic device. The processor 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0112] Typically, the following devices can be connected to I / O interface 705: input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709. Communication device 709 allows electronic devices to exchange data via wireless or wired communication with other devices. Although Figure 7 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0113] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 709, or installed from a memory 708, or installed from a ROM 702. When the computer program is executed by the processor 701, it performs the functions defined in the methods of the embodiments of the present invention.

[0114] Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0115] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0116] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0117] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A data production method, characterized in that, The method, applied to a production platform with schedulable cloud resources, includes: Acquire raw sensor data; Obtain the corresponding task configuration template according to the autonomous driving training task. The task configuration template is a template that is pre-configured and stored based on the user's production requirements for data. The production requirements correspond to the autonomous driving training task. The original sensing data and the task configuration template are bound together to generate the current task instance; Based on the type of the current task instance, the current task instance is deployed to the corresponding target ordinary POD or target persistent POD for execution. The target ordinary POD is created when the task is deployed and releases cloud resources when the task ends. The target persistent POD is created when the task is deployed for the first time and releases cloud resources after receiving a stop command and all internally running tasks are completed. Obtain the training data obtained after the current task instance is executed.

2. The method according to claim 1, characterized in that, The steps for configuring the task configuration template include: Configure data processing version information, resource and environment configuration information, processing link parameters, and task management identifier. The data processing version information is used to specify the operator version used for data filtering and processing. The resource and environment configuration information is used to define the hardware resource information for task execution. The processing link parameters are used to represent the execution parameters required by the data processing operators. The task management identifier is used to distinguish task types and mark task priorities.

3. The method according to claim 1, characterized in that, The step of binding the original sensing data and the task configuration template to generate the current task instance includes: If the original sensing data is a single data point, then the original sensing data and the task configuration template are directly bound together to generate the current task instance; If the original sensing data is batch data, then the original sensing data is divided into multiple independent data units; Each data unit is bound to the task configuration template to generate multiple corresponding current task instances.

4. The method according to claim 1, characterized in that, The step of dispatching the current task instance to the corresponding target ordinary POD or target resident POD for execution based on the type of the current task instance includes: If the current task instance is defined as a normal task, then when the idle cloud resources meet the availability conditions of the current task instance, a target normal POD is created for the current task instance; The current task instance is deployed to the target ordinary POD for execution, and the target ordinary POD is released when the current task instance finishes execution. If the current task instance is defined as a persistent task, determine whether there is a first task instance that was issued and executed before the current task instance, and the first task instance has the same attribute information as the current task instance; If it does not exist, when the idle cloud resources meet the availability conditions of the current task instance, a first target persistent POD is created for the current task instance, and the current task instance is sent to the first target persistent POD for execution. If it exists, then find the second target persistent POD that is running the first task instance, and when the second target persistent POD has the resource conditions to receive the current task instance, send the current task instance to the second target persistent POD for execution.

5. The method according to claim 4, characterized in that, The steps for performing internal tasks through the target ordinary POD or the target resident POD include: Split all task instances within the current POD into multiple execution nodes according to the smallest execution unit; Based on the execution order of each execution node and the type of hardware resources used, multiple execution nodes that do not conflict are selected to execute concurrently within the current POD.

6. The method according to claim 4, characterized in that, When idle cloud resources meet the availability conditions of the current task instance, the steps for creating the target ordinary POD or the target resident POD for the current task instance include: The number of tasks that can be issued in this cycle is determined based on the current idle cloud resources; The task instances are sorted based on the priority of the current task instance and the priority of other task instances within the current period, and it is determined whether the current task instance is within the number of tasks. If the current task instance is within the number of tasks, then create the target ordinary POD or the target resident POD for the current task instance, and update the current idle cloud resources.

7. The method according to claim 6, characterized in that, When multiple task instances need to be deployed, they are sorted in descending order of priority as follows: The task configuration template used by the task instance to be issued has a preset number of dedicated cloud resources; the task configuration template used by the task instance to be issued has a preset number of shared cloud resources; if the dedicated cloud resources and the shared cloud resources are already occupied, the task configuration template used by the task instance to be issued has higher priority. When template priorities are the same, the task instance to be issued has a higher priority.

8. The method according to claim 5, characterized in that, The method further includes: Get the POD status reported by each currently running resident POD; Based on the POD status, determine whether the number of resident PODs under any task configuration template is greater than the maximum number of PODs that can be opened in parallel for that task configuration template. When the number of resident PODs under any task configuration template exceeds the limit of parallel PODs that can be enabled in that task configuration template, the redundant PODs of that task configuration template will be selected and shut down based on the number of execution nodes in each POD.

9. The method according to claim 1, characterized in that, The stop command is issued when there are no remaining tasks waiting to be processed in the task configuration template corresponding to the target resident POD.

10. A data production apparatus, characterized in that, The apparatus, used in a production platform with schedulable cloud resources, includes: The raw sensing data acquisition module is used to acquire raw sensing data; The template acquisition module is used to acquire the corresponding task configuration template according to the autonomous driving training task. The task configuration template is a template that is pre-configured and stored based on the user's production requirements for data. The production requirements correspond to the autonomous driving training task. The task generation module is used to bind the raw perception data and the task configuration template to generate the current task instance; The resource scheduling production module is used to distribute the current task instance to the corresponding target ordinary POD or target resident POD for execution based on the type of the current task instance. The target ordinary POD is used to create when the task is distributed and release cloud resources when the task ends. The target resident POD is used to create when the task is distributed for the first time and release cloud resources after receiving a stop command and all internally running tasks are completed. The data output module is used to obtain the training data obtained after the current task instance is executed.

11. An electronic device, characterized in that, include: A memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, the processor executing the computer instructions to perform the method of any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the method of any one of claims 1 to 9.

Images