Data task processing method and device, storage medium and electronic equipment

Abstract

The invention discloses a data task processing method and device, a storage medium and electronic equipment. The method comprises the steps of determining operators matched with execution steps in a workflow of a data processing task according to task description information of the data processing task; according to operator operation modes corresponding to the operators matched with the execution steps, a workflow operation mode matched with the workflow is determined, and the operator operation modes are used for indicating data access modes when the operators execute the data processing task; instantiating operators matched with the execution steps according to a workflow operation mode to obtain operator containers matched with the multiple operators respectively; and calling the corresponding operator container based on the execution sequence of each execution step in the workflow to obtain a task execution result matched with the data processing task. The technical problem of low processing efficiency of complex data tasks in related technologies is solved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a data task processing method, apparatus, storage medium, and electronic device. Background Technology

[0002] Autonomous driving generates massive amounts of data during various data collection and testing tasks. This data undergoes processing, labeling, and quality control in the cloud, ultimately forming high-value data for iterative optimization and testing of autonomous driving algorithms. As the scale of data and the complexity of autonomous driving operations increase, the demands for data processing complexity and timeliness also rise.

[0003] Currently, existing technologies typically use an operator-workflow approach, abstracting each data processing module into an independent operator module and then connecting them using a directed acyclic graph (DAG) to form a specific data processing workflow. However, when handling complex tasks, especially those involving the reuse of output data from previous operators by different operators, the existing serial processing chain only has a single, fixed dependency relationship between adjacent operators, failing to achieve cross-operator data reuse. This results in multiple rounds of repeated data processing and transmission, reducing the execution efficiency of data processing tasks and increasing the resources consumed in executing them. In other words, existing technologies suffer from low processing efficiency for complex data tasks. Summary of the Invention

[0004] This application provides a data task processing method, apparatus, storage medium, and electronic device to at least solve the technical problem of low processing efficiency for complex data tasks in the prior art.

[0005] According to one aspect of the embodiments of this application, a data task processing method is provided, comprising: determining, based on task description information of the data processing task, operators matching each execution step in the workflow of the data processing task, wherein the task description information is used to indicate the data processing logic of the data processing task; determining a workflow operation mode matching the workflow based on the operator operation mode corresponding to each operator matching each execution step, wherein the operator operation mode is used to indicate the way the operator accesses data when executing the data processing task; instantiating the operators matching each execution step according to the workflow operation mode to obtain operator containers matching multiple operators, wherein the operator containers include execution code corresponding to the execution steps and environment variables; and calling the corresponding operator containers based on the execution order of each execution step in the workflow to obtain a task execution result matching the data processing task.

[0006] According to another aspect of the embodiments of this application, a data task processing apparatus is also provided, comprising: a first determining unit, for determining operators matching each execution step in the workflow of the data processing task based on task description information of the data processing task, wherein the task description information is used to indicate the data processing logic of the data processing task; a second determining unit, for determining a workflow operation mode matching the workflow based on the operator operation mode corresponding to each operator matching each execution step, wherein the operator operation mode is used to indicate the way the operator accesses data when executing the data processing task; an instantiation unit, for instantiating the operators matching each execution step according to the workflow operation mode to obtain operator containers matching each of the multiple operators, wherein the operator containers include execution code and environment variables corresponding to the execution steps; and a processing unit, for calling the corresponding operator containers based on the execution order of each execution step in the workflow to obtain a task execution result matching the data processing task.

[0007] As an optional solution, the second determining unit further includes: a third determining module, used to determine the current storage directory corresponding to the current input data set of the operator under the current execution step; and to determine the operator operation mode of the operator under the current execution step according to the relationship between the directory corresponding to the first output data set of the operator under the first reference execution step and the current storage directory, wherein the first reference execution step is the previous execution step of the current execution step.

[0008] As an optional solution, the aforementioned third determining module includes: a fourth determining module, used to determine, when the current storage directory is the directory corresponding to the first output data set, that the operator running mode of the operator matching the current execution step is serial mode; and when the current storage directory is the directory corresponding to the first output data set and the second output data set, and the current input data set includes the first output data set and the second output data set, to determine that the operator running mode of the operator matching the current execution step is shared mode, wherein the second output data set is the output data set of the operator under the second reference execution step, and the second reference execution step is the execution step before the first reference execution step.

[0009] As an optional solution, the aforementioned third determining module includes: a fifth determining module, used to determine the workflow operation mode as a serial operation mode when the operator operation mode of each execution step in the workflow is a serial mode; to determine the workflow operation mode as a shared operation mode when the operator operation mode of each execution step in the workflow is a shared mode; and to determine the workflow operation mode as a mixed operation mode when the workflow contains both serial mode operators and shared mode operators.

[0010] As an optional solution, the fifth determining module mentioned above includes: a sixth determining module, used to determine the input path and output path that match multiple operators in the workflow when the workflow operation mode is serial operation mode, wherein the operator obtains data in the first storage directory based on the input path, and the operator writes data to the second storage directory based on the output path; configure read-only permissions for the first storage directory corresponding to the input path in the environment variables corresponding to each of the multiple operators, and configure read-write permissions for the second storage directory corresponding to the output path in the environment variables; and instantiate based on the environment variables corresponding to each of the multiple operators to obtain the operator container that matches each of the multiple operators.

[0011] As an optional solution, the fifth determining module mentioned above also includes a seventh determining module, which is used to determine the shared path matching all operators in the workflow when the workflow operation mode is the shared operation mode. Here, the operator obtains data in the shared storage directory based on the shared path, and the operator writes the data to the shared storage directory based on the shared path; read and write permissions are configured for the shared storage directory corresponding to the shared path in the environment variables of each operator; and an operator container matching each operator is obtained based on the environment variables of each operator.

[0012] As an optional solution, the fifth determining module mentioned above also includes an eighth determining module, used to configure read-only permissions for the first storage directory corresponding to the input path in the environment variables of the serial mode operator when the workflow operation mode is a hybrid operation mode, configure read-write permissions for the second storage directory corresponding to the output path in the environment variables, and configure read-write permissions for the shared storage directory corresponding to the shared path in the environment variables of the shared mode operator; add a copy operator between the first reference operator in the serial mode and the second reference operator in the shared mode, which have an adjacent position relationship, wherein the copy operator is used to copy the data in the second storage directory corresponding to the output path of the first reference operator to the shared storage directory corresponding to the shared path of the second reference operator; and instantiate based on the environment variables of each operator and copy operator in the workflow to obtain an operator container that matches each operator and copy operator in the workflow.

[0013] As an optional solution, the first determining unit includes a configuration module for configuring static environment variables for operators of each execution step, wherein the static environment variables are used to indicate the type of configuration parameters; when a workflow is triggered, it sequentially obtains the parameter transmission data information corresponding to each execution step; based on the parameter transmission data information, it configures dynamic environment variables for operators corresponding to each execution step, wherein the dynamic environment variables are used to indicate the type of transmission parameters; when calling the corresponding operator container based on the execution order of each execution step in the workflow, it configures transitive environment variables for the operator corresponding to the second execution step based on the execution result obtained from calling the operator container corresponding to the first execution step, wherein the execution order of the first execution step is before the second execution step, the priority of the transitive environment variable is greater than the priority of the dynamic environment variable, and the priority of the dynamic environment variable is greater than the priority of the static environment variable.

[0014] As an optional solution, the above processing unit is also used to determine the target execution steps corresponding to at least one target operator when there is at least one target operator of a target type among multiple operators; when the target execution step is executed, call the monitoring interface to query the operation status of the target operation; and when the operation status is a completed state, call the execution steps after the target execution step.

[0015] According to another aspect of the embodiments of this application, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the data task processing method described above.

[0016] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor performs the above-described data task processing method through the computer program.

[0017] Through the above-described implementation method of this application, based on the task description information of the data processing task, operators matching each execution step in the workflow of the data processing task are determined. By parsing the task description information, the specific data processing requirements and logic of each step in the workflow are clarified, and the complex task is subdivided into a series of independent operators that are easy to manage and execute. Furthermore, the operating mode (serial, shared) of the operators is identified, and a suitable workflow operating mode is selected for the entire workflow. Based on the workflow operating mode, the operators matching each execution step are instantiated to obtain operator containers that match each of the multiple operators. Each operator container runs independently and can dynamically adjust resource allocation according to its own needs, reducing resource conflicts and waste. Finally, based on the actual execution process of the workflow, the operator containers are called according to the execution order of each step to ensure the continuity and logic of data processing. The orderly calling of operator containers realizes the correct flow and processing of data in the entire processing chain, solving the technical problem of low processing efficiency for complex data tasks in related technologies. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 This is a flowchart of an optional data task processing method according to an embodiment of this application;

[0020] Figure 2 This is a schematic diagram of an optional data task processing method according to an embodiment of this application;

[0021] Figure 3 This is a schematic diagram of an optional workflow operation mode according to an embodiment of this application;

[0022] Figure 4 This is a schematic diagram of another optional workflow operation mode according to an embodiment of this application;

[0023] Figure 5 This is a schematic diagram of another optional workflow operation mode according to an embodiment of this application;

[0024] Figure 6 This is a schematic diagram of an optional environment variable configuration according to an embodiment of this application;

[0025] Figure 7 This is a schematic diagram illustrating another optional environment variable configuration according to an embodiment of this application;

[0026] Figure 8 This is a schematic diagram of another optional data task processing method according to an embodiment of this application;

[0027] Figure 9 This is a schematic diagram of another optional data task processing method according to an embodiment of this application;

[0028] Figure 10 This is a schematic diagram of a data task processing apparatus according to an embodiment of this application;

[0029] Figure 11 This is a schematic diagram of the structure of an optional electronic device according to an embodiment of this application. Detailed Implementation

[0030] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0031] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0032] As an optional implementation method, such as Figure 1 As shown, the data processing method can be executed by electronic devices, and the specific steps include:

[0033] S102, Based on the task description information of the data processing task, determine the operators in the workflow of the data processing task that match each execution step, wherein the task description information is used to indicate the data processing logic of the data processing task.

[0034] S104, determine the workflow operation mode that matches the workflow based on the operator operation mode corresponding to each operator that matches each execution step, wherein the operator operation mode is used to indicate the way the operator accesses data when performing data processing tasks;

[0035] S106, instantiate the operators that match each execution step according to the workflow operation mode to obtain the operator container that matches each of the multiple operators. The operator container includes the execution code and environment variables corresponding to the execution step.

[0036] S108: Based on the execution order of each execution step in the workflow, call the corresponding operator container to obtain the task execution result that matches the data processing task.

[0037] In S102 of the above embodiment, operators matching each execution step in the workflow of the data processing task are determined based on the task description information of the data processing task. The task description information is used to indicate the data processing logic of the data processing task. The data processing task includes, but is not limited to, data labeling, data parsing, and algorithm prediction. The task description information is, for example, a specification of the data processing task, including task objectives, data format, processing logic, operator requirements, etc. The workflow is an ordered set of a series of operators used to organize and execute the data processing task. The execution steps are each independent data processing stage in the workflow.

[0038] As an optional implementation, a task description file defines the detailed logic of the data processing task. This file may contain information such as the type of input data, the processing order, and the expected output results. After parsing the task description file, the system automatically matches operators from the operator library, ensuring that each operator meets the corresponding step requirements in the task description. For example, for a task involving data cleaning, feature extraction, and model training, the system will match cleaning operators, feature extraction operators, and training operators respectively, and construct the workflow in sequence.

[0039] In step S104 above, the workflow operation mode matching the workflow is determined according to the operator operation mode corresponding to each operator that matches each execution step. The operator operation mode is used to indicate the way the operator accesses data when performing data processing tasks.

[0040] Optionally, the above-mentioned operator operation modes include serial mode, shared mode, and mixed mode, indicating how the operator reads and writes data and interacts with other operators; the above-mentioned workflow operation mode is the execution method of the entire data processing task determined based on the operator operation mode.

[0041] As an optional implementation, if all operators use serial mode, the workflow will be configured for serial execution; if operators in shared mode exist, the system will configure the workflow in shared mode; if it is a combination of operators in mixed mode, the workflow will be set to mixed mode. For example, when a feature extraction operator needs to read information from multiple data sources, the system identifies the operator's operating mode as shared mode.

[0042] In step S106 above, operators matching each execution step are instantiated according to the workflow operation mode to obtain operator containers that match each of the multiple operators. The operator container includes the execution code corresponding to the execution step and environment variables. The operator container refers to a container instantiated for each operator in the K8S system for independent operation of the operator. The execution code is the algorithm code inside the operator. The environment variables are the configuration parameters required for the operator to run, including static, dynamic and transitive environment variables.

[0043] In some alternative implementations, within a Kubernetes cluster, the workflow engine instantiates each operator as an independent Pod (operator container) and injects static, dynamic, and transitive environment variables according to the operator's runtime requirements. For example, for a data parsing operator, the system configures the source and output paths of the input data, as well as possible format conversion parameters, within the operator container.

[0044] In step S108 above, the corresponding operator container is called based on the execution order of each execution step in the workflow to obtain the task execution result that matches the data processing task.

[0045] As an optional implementation, the workflow engine starts the operator containers one by one according to the execution order of the workflow by calling the operator container's startup interface. For example, the data parsing operator container is called first, followed by the feature extraction operator container, and finally the model training operator container. The output of each operator container serves as the input of the subsequent operator container until the entire workflow is completed and the final data processing result is generated.

[0046] As per the process Figure 2 As shown, the workflow includes operator 1, operator 2 and operator 3, which are instantiated as pod1, pod2 and pod3 in Kubernetes. The scheduler calls them in sequence, and retryes when it fails (no more than twice). After the call is completed, the container is destroyed and the data in the storage directory of the PV (persistent volume) is cleaned up at regular intervals.

[0047] Through the above-described implementation method of this application, based on the task description information of the data processing task, operators matching each execution step in the workflow of the data processing task are determined. By parsing the task description information, the specific data processing requirements and logic of each step in the workflow are clarified, and the complex task is subdivided into a series of independent operators that are easy to manage and execute. Furthermore, the operating mode (serial, shared) of the operators is identified, and a suitable workflow operating mode is selected for the entire workflow. Based on the workflow operating mode, the operators matching each execution step are instantiated to obtain operator containers that match each of the multiple operators. Each operator container runs independently and can dynamically adjust resource allocation according to its own needs, reducing resource conflicts and waste. Finally, based on the actual execution process of the workflow, the operator containers are called according to the execution order of each step to ensure the continuity and logic of data processing. The orderly calling of operator containers realizes the correct flow and processing of data in the entire processing chain, solving the technical problem of low processing efficiency for complex data tasks in related technologies.

[0048] In one optional implementation, before determining the workflow execution mode matching the workflow based on the operator execution mode corresponding to each operator matching the respective execution step, the following steps are included:

[0049] S1, determine the current storage directory corresponding to the current input data set of the operator in the current execution step;

[0050] S2, based on the relationship between the directory corresponding to the first output data set of the operator under the first reference execution step and the current storage directory, determine the operator operation mode of the operator in the current execution step, wherein the first reference execution step is the previous execution step of the current execution step.

[0051] Optionally, in steps S1-S2 above, the operator operation mode of the current operator is determined based on the relationship between the input data source of the current operator and the output data storage directory of the first reference execution step. If the current operator only depends on the output data of the first reference execution step, then the current operator is as follows: Figure 3 The serial mode shown in operator 2; if the current operator needs to access the output data of multiple previous execution steps, then the current operator is as follows: Figure 4 The sharing mode shown in operator 2.

[0052] As an optional implementation, if the current execution step is data post-processing, and its input data comes from the data parsing and feature extraction steps, the current operator is identified as being in shared mode. The system will set the output directory corresponding to the data parsing and feature extraction steps as the shared storage directory for the operator of the current execution step, in order to achieve efficient data reuse and cross-step access.

[0053] Through the above-described embodiments of this application, the current storage directory corresponding to the current input data set of the operator in the current execution step is determined, ensuring that each operator can accurately obtain the input data required for its processing, thus optimizing the data transmission process; based on the relationship between the directory corresponding to the first output data set of the operator in the first reference execution step and the current storage directory, the operator operation mode of the operator in the current execution step is determined, and different operation modes (such as serial and shared) correspond to different data processing logic and resource allocation strategies, thereby improving the efficiency of processing complex data processing tasks.

[0054] In one optional implementation, the operator operation mode of the operator in the current execution step is determined based on the relationship between the directory corresponding to the first output data set of the operator under the first reference execution step and the current storage directory, including:

[0055] S1, if the current storage directory is the directory corresponding to the first output data set, determine that the operator running mode of the operator matching the current execution step is serial mode;

[0056] S2, if the current storage directory is the directory corresponding to the first output data set and the second output data set, and the current input data set includes the first output data set and the second output data set, determine that the operator running mode of the operator matching the current execution step is the shared mode, wherein the second output data set is the output data set of the operator under the second reference execution step, and the second reference execution step is the execution step before the first reference execution step.

[0057] In step S1 above, if the input dataset of the current operator comes from the output of the previous operator, that is, if the input data storage directory of the current operator matches the output directory of the previous operator, then the operator's operation mode is determined to be serial mode. Figure 3 As shown in operator 2, the data comes only from Data2.

[0058] In step S2 above, when an operator's input dataset comes not only from the output of the immediate preceding operator but also from the output of even earlier operators—that is, when the current operator needs to access data from two or more upstream operators simultaneously—the current operator's operating mode is determined to be shared mode to ensure cross-operator sharing and reuse of data. Figure 4 As shown in operator 2, the data comes from Data1 and Data2. It should be noted that the operators corresponding to the first output data set and the operators corresponding to the second output data set are some or all of the operators with upstream and downstream dependencies in the workflow. They do not represent the first and second in the workflow sequence, but are determined according to the actual situation.

[0059] For example, the current execution step is data fusion. The input data for the fusion operator comes not only from the output of the feature extraction operator (the first output data set) but also from the output of the data parsing operator (the second output data set). Recognizing that the fusion operator needs to access data from two different upstream operators, the fusion operator's operating mode is configured to shared mode. That is, the fusion operator has a shared storage directory, such as / mnt / shared_data, which will contain the output of the feature extraction operator ( / mnt / processed_data) and the output of the data parsing operator ( / mnt / origin_data), for the fusion operator to access and use.

[0060] The above method dynamically determines the operator's operating mode based on the relationship between the storage directories where the input and output data of different operators are located. It determines whether the current operator needs to share data with other operators or can run independently, ensuring accurate data reading, optimizing data transmission and resource sharing, and thus improving data processing efficiency.

[0061] In one optional implementation, a workflow execution mode matching the workflow is determined based on the operator execution mode corresponding to each operator matching the respective execution step, including:

[0062] S1, when the operator operation mode of each execution step in the workflow is serial mode, the workflow operation mode is determined to be serial operation mode.

[0063] When all operators in the workflow run in a serial mode, meaning that the input data for each operator comes only from the output of its immediate preceding operator, such as... Figure 3 As shown, the entire workflow should be set to serial operation mode, which means that the entire data processing flow will be executed sequentially, with the next operator only starting to run after one operator is completed.

[0064] S2, when the operator operation mode of each execution step in the workflow is the shared operation mode, determine the workflow operation mode as the shared operation mode;

[0065] When all operators in a workflow are running in shared mode, it indicates that operators can share one or more sets of datasets generated by earlier operators, rather than relying solely on the output of the directly preceding operator. Figure 4 As shown. At this point, the entire workflow should be set to a shared operation mode, allowing free access to data and parallel processing between operators. For example, operator 5 can obtain the output data of operators 1, 2, and 4.

[0066] For example, in autonomous driving data processing, a series of operators need to access and analyze the same set of sensor data. Multiple feature extraction operators, for instance, do not directly depend on each other's outputs but rather on the output of the data parsing operator. In this case, defining the entire workflow as a shared operation mode means that operators can run in parallel, accessing the same shared dataset simultaneously, thus improving processing efficiency. Specifically, all operators will share the same storage directory, such as ` / mnt / shared_data`, which contains the output data of the data parsing operator. Multiple feature extraction operators can read data from this directory in parallel and perform independent processing activities.

[0067] S3, when the workflow contains both serial mode operators and shared mode operators, determines the workflow operation mode as a mixed operation mode.

[0068] When a workflow contains both serial and shared operators, the overall workflow operation mode should be identified as a hybrid operation mode, such as... Figure 5 As shown, this means that the data processing flow will include a combination of linear sequential execution and data-sharing parallel execution to accommodate more complex and diverse data processing needs.

[0069] For example, in an autonomous driving data closed loop, the data parsing operator may generate raw sensor data, the feature extraction operator processes the data based on the output of the data parsing operator (serial mode), while the data verification operator and the anomaly detection operator depend not only on the output of the feature extraction operator but also on the raw data (shared mode). In this case, the workflow operation mode will be set to a hybrid mode. Specifically, the feature extraction operator is executed serially after the data parsing operator, and its output is stored in a directory in the serial mode, such as / mnt / output_serial; while the data verification and anomaly detection operators will share the raw data and the output of the feature extraction operator. They can read the required data from a shared storage directory (such as / mnt / shared_data) and perform parallel or on-demand processing.

[0070] Determining the workflow operation mode allows the system to choose between serial, shared, and hybrid modes, thereby rationally allocating computing and storage resources and reducing resource idleness and waste. Serial mode ensures the sequentiality of data processing and reduces the uncertainty of data transmission; shared mode reduces the frequency of data copying and transmission through centralized storage management, accelerating the data processing flow; and hybrid mode can flexibly handle the needs of different data processing stages and balance data access and processing speed.

[0071] In one optional implementation, operators matching each execution step are instantiated according to the workflow operation mode to obtain operator containers matching each of the multiple operators, including:

[0072] S1, when the workflow operation mode is serial operation mode, determine the input path and output path that match the multiple operators in the workflow respectively, wherein the operator obtains data in the first storage directory based on the input path, and the operator writes the data to the second storage directory based on the output path.

[0073] In a serial workflow, each operator is assigned a specific input path and output path. The input path points to the output directory of the previous operator, while the output path is the input directory of the downstream operator of the current operator. For example, the input path of the feature extraction operator points to the output directory of the data preprocessing operator, and the output path of the feature extraction operator is the input directory of the data fusion operator.

[0074] S2, configure read-only permissions for the first storage directory corresponding to the input path in the environment variables of each of the multiple operators, and configure read-write permissions for the second storage directory corresponding to the output path in the environment variables;

[0075] S3 instantiates based on the environment variables corresponding to each of the multiple operators, resulting in operator containers that match each operator. The workflow engine traverses the entire workflow and creates Pods (operator containers) using the Kubernetes API or kubectl command based on the environment variable configuration of each operator. In the Pod configuration, the mount source and permissions of the / mnt / input and / mnt / output directories are precisely specified through the definitions of volumeMounts and volumes. In addition, the startup parameters and resource configuration of the operator container will also be customized according to the environment variables. For example, for feature extraction operators, their operator containers may be configured to have higher CPU and memory resources to accelerate data processing.

[0076] As an optional implementation, the operator's environment variables will by default include two paths: the input directory and the output directory, as well as other parameters such as the running mode. Each operator will retrieve data from a fixed path within the container and write the data to another output path within the container. The fixed input and output paths within the container will be mounted to the actual input and output paths in the PV through the actual runtime environment variable configuration, and will be assigned the corresponding permissions.

[0077] The input directory defaults to / mnt / input / within the container, and this directory is mounted to the actual input directory. This directory is set to read-only permissions, and the algorithm obtains relevant input data from this directory by default. The output directory defaults to / mnt / output / within the container, and this directory is mounted to the actual output directory. This directory has read and write permissions, and the algorithm places the output products in this directory by default.

[0078] like Figure 3 As shown, in this mode, the input path of each serial operator is the output path of the previous running operator, and similarly, the output path of the operator will become the input path of the downstream adjacent operator.

[0079] By configuring permissions, it is ensured that operators in serial mode can only read their input data during their execution phase, and cannot modify or delete it. At the same time, operators are allowed to write data in their own output paths. Furthermore, operators are instantiated into containers to encapsulate and isolate them, and it is ensured that each operator has the environment configuration and resources required for its execution, thereby improving the processing efficiency of complex data tasks.

[0080] In one optional implementation, operators matching each execution step are instantiated according to the workflow operation mode to obtain operator containers matching each of the multiple operators, including:

[0081] S1, when the workflow is in shared operation mode, determine the shared path that matches all operators in the workflow, where operators obtain data from the shared storage directory based on the shared path, and write the data to the shared storage directory based on the shared path.

[0082] In the shared operation mode described above, all operators that need to share data access the same shared path, which is mapped to a shared storage directory within the Kubernetes cluster. Operators can read data from this directory and also write their output results to it, allowing multiple operators to read and write the same set of data simultaneously. This promotes efficient data reuse in the process without the need for additional data copying or transmission.

[0083] S2 configures read and write permissions for the shared storage directory corresponding to the shared path in the environment variables of each operator.

[0084] Before instantiating the operator container, the correct access permissions need to be configured for the environment variables of each operator to ensure that the operators can correctly read and write to the shared storage directory. Specifically, the shared storage directory should be configured with read and write permissions in the operator container so that all operators in shared mode can perform data read and write operations.

[0085] S3 instantiates based on the environment variables of each operator, resulting in an operator container that matches each operator.

[0086] Based on the environment variable configuration of operators, including shared paths and their permission settings, the Kubernetes system instantiates each operator as an independent container (Pod), and each container can access the specified shared storage directory according to its environment variables to perform data reading and writing operations.

[0087] As an optional implementation, in the shared operation mode, the data read by the downstream operator no longer comes only from the adjacent upstream, but from data generated further upstream. For example, in the data annotation process, the downstream (post-processing) operator not only needs to obtain the algorithm results of the adjacent upstream (NN model inference), but also needs to obtain the raw parsing data of the further upstream (data parsing) in order to optimize the annotation results. In this process, the upstream data will be frequently used by multiple downstream operators.

[0088] In the above scenario, the operator's environment variables will include a path to a shared read / write directory, as well as other parameters such as the operating mode. Each operator will read data from a fixed path within the container and write the data to that same fixed path. This fixed path within the container will be mounted to a shared directory in the PV through the actual runtime environment variable configuration, making it accessible to all operators in the workflow and granting them corresponding permissions.

[0089] The shared directory is set to / mnt / share / within the container by default, and this directory is mounted to the actual shared storage directory in the physical PV. Figure 4 As shown. This directory has read and write permissions and can be accessed by operators in the same shared mode within the same workflow. In this mode, each operator in the shared mode can access this path and write its output.

[0090] In shared operation mode, by providing a unified shared path for all operators, configuring read and write permissions, and instantiating operators as containers, the data processing efficiency of complex workflows is significantly improved. The use of shared paths reduces data transmission overhead, the configuration of read and write permissions promotes efficient collaboration between operators, and operator containerization ensures the speed and independence of operator execution. Overall, this not only improves data processing speed but also enhances the stability and adaptability of the workflow.

[0091] In one optional implementation, operators matching each execution step are instantiated according to the workflow operation mode to obtain operator containers matching each of the multiple operators, including:

[0092] S1, when the workflow operation mode is a hybrid operation mode, configure read-only permissions for the first storage directory corresponding to the input path in the environment variables of the serial mode operator, configure read-write permissions for the second storage directory corresponding to the output path in the environment variables, and configure read-write permissions for the shared storage directory corresponding to the shared path in the environment variables of the shared mode operator.

[0093] In other words, in the hybrid operation mode, different permissions are configured for operators in serial mode and shared mode. The input directory (first storage directory) of serial mode operators is set to read-only to ensure data consistency and integrity, while the output directory (second storage directory) requires read and write permissions so that operators can store processing results; the shared directory (shared storage directory) of shared mode operators is configured with read and write permissions, allowing multiple operators to access and update it.

[0094] S2, add a copy operator between the first reference operator in serial mode and the second reference operator in shared mode, which have adjacent positional relationship. The copy operator is used to copy the data in the second storage directory corresponding to the output path of the first reference operator to the shared storage directory corresponding to the shared path of the second reference operator.

[0095] For example, when transitioning from a serial mode operator to a shared mode operator, a dedicated copy operator is needed. This operator reads data from the output directory of the serial mode operator and copies it to the shared storage directory for use by subsequent shared mode operators. This ensures that shared mode operators can access the data processed by the serial mode operators without disrupting the original data flow order. Similarly, when transitioning from a shared mode operator to a serial mode operator, a copy operator copies the stored data from the shared storage directory to the first storage directory corresponding to the first adjacent serial mode operator.

[0096] S3 instantiates an operator container based on the environment variables of each operator and copy operator in the workflow, resulting in an operator container that matches each operator and copy operator in the workflow. In hybrid operation mode, each operator in the workflow (including operators in serial mode, operators in shared mode, and copy operators used for data transformation) is instantiated independently based on its own environment variables, resulting in a corresponding operator container.

[0097] As an optional implementation method, in such Figure 5 In the hybrid operation mode shown, operators in both serial and shared modes exist in the workflow. During the switching process, a special operator is automatically introduced to copy data. Specifically, if it is from an upstream serial operator to a downstream shared operator, the special operator copies the output path data of the upstream serial operator in the PV to the shared storage path for the downstream shared operator to obtain. If it is from an upstream shared operator to a downstream serial operator, the special operator copies the shared path data in the corresponding PV of the upstream shared operator to the new path as the input path of the downstream serial operator.

[0098] In hybrid operation mode, by configuring permissions for operators in different modes, introducing copy operator processing mode conversion, and instantiating all operators as containers, the problem of low data processing efficiency in complex workflows is solved. Permission configuration ensures the security and orderliness of data processing, copy operators accelerate data conversion and transmission, and operator containerization further improves the efficiency of operator execution and the flexibility of resource management. Overall, this improves the execution speed of the workflow and the reliability of data processing, providing an efficient and stable data processing framework for high-volume and high-complexity data processing scenarios.

[0099] In one optional implementation, after determining the operators matching each execution step in the workflow of the data processing task based on the task description information of the data processing task, the process includes:

[0100] S1, Configure static environment variables for operators of each execution step, wherein the static environment variables are used to indicate the type of configuration parameters;

[0101] S2, when the workflow is triggered, sequentially obtain the parameter data information corresponding to each execution step;

[0102] S3, configure dynamic environment variables for the operators corresponding to each execution step based on the parameter data information, wherein the dynamic environment variables are used to indicate the type of the sent parameters;

[0103] S4. When calling the corresponding operator container based on the execution order of each execution step in the workflow, based on the execution result obtained by calling the operator container corresponding to the first execution step, configure transitive environment variables for the operator corresponding to the second execution step. Here, the execution order of the first execution step is before the second execution step, the priority of transitive environment variables is greater than the priority of dynamic environment variables, and the priority of dynamic environment variables is greater than the priority of static environment variables.

[0104] In step S1 above, static environment variables are configured for the operators of each execution step. The static environment variables are used to indicate the type of configuration parameters. The static environment variables are parameters set in the operator configuration stage and remain unchanged throughout the entire workflow lifecycle. They are used to guide the basic behavior of the operators, such as workflow type, operator running mode (shared, serial, etc.) or specific notification address (such as the address of WeChat Work robot). They are determined when the workflow is created and do not need to be changed every time the workflow runs.

[0105] In steps S2-S3 above, when the workflow is triggered, the parameter data information corresponding to each execution step is obtained sequentially; dynamic environment variables are configured for the operators corresponding to each execution step based on the parameter data information, wherein the dynamic environment variables are used to indicate the type of the sent parameters;

[0106] When a workflow is triggered, the workflow engine receives parameter data from external sources via API calls or other means, such as the specific path to the file to be processed, a specific timestamp, or real-time status. These values ​​may differ each time the workflow runs. Then, based on the order of the data flow, the engine converts this information into dynamic environment variables and passes them to the operator container that will be running. For example, when starting a workflow that includes data preprocessing and feature extraction, the workflow engine receives a parameter containing the path to the original data file. It then passes this path as a dynamic environment variable (such as INPUT_DATA_PATH) to the data preprocessing operator container, ensuring that the operator can access the correct input at runtime.

[0107] In step S4 above, when calling the corresponding operator container based on the execution order of each execution step in the workflow, a transitive environment variable is configured for the operator corresponding to the second execution step based on the execution result obtained by calling the operator container corresponding to the first execution step. The execution order of the first execution step is before the second execution step, the priority of the transitive environment variable is greater than the priority of the dynamic environment variable, and the priority of the dynamic environment variable is greater than the priority of the static environment variable.

[0108] For example, in the autonomous driving data processing workflow, after the data preprocessing (first execution step) operator completes data cleaning and format conversion, it writes the processing results (such as parameters for a specific filename or data format) to a .env file located in a shared storage directory or output directory. Subsequently, when the feature extraction operator (second execution step) starts, the workflow engine automatically reads this .env file, retrieves the execution results from it, and injects them as transitive environment variables into the feature extraction operator's container. That is, the feature extraction operator can process data based on the latest output of the data preprocessing operator.

[0109] As an optional implementation method, workflow operation requires certain environmental variable configurations. Based on the impact of different environmental variables on the workflow, these environmental variables are divided into three layers:

[0110] like Figure 6 As shown, static environment variables refer to the parameter types that can be configured during the workflow configuration phase of an operator. This type of environment variable will not change when multiple instances of the same workflow are created. For example, in the WeChat Work notification operator, the corresponding chatbot address can be configured during the workflow configuration phase, thereby enabling different workflows to send messages to different WeChat groups based on different parameters of the same operator.

[0111] Dynamic environment variables refer to the types of parameters sent to the workflow by the operator during the workflow triggering phase. These environment variables change with each instantiation. For example, the folder download operator uses dynamic parameters to obtain the data path to be processed during the current workflow execution, thus enabling the same workflow to process different data. Static and dynamic environment variables are obtained and used in the same way within the operator; they only affect how the workflow obtains parameters.

[0112] like Figure 7 As shown, transitive environment variables refer to the results processed by upstream operators during the actual processing phase of a workflow, which are then passed to downstream operators as environment variables for loading. These environment variables cannot be determined at the beginning of workflow creation; they depend on the processing results. For example, after upstream operators perform data analysis and determine that a certain topic corresponds to the actual recorded images, downstream data mining operators need to obtain the images from the data. However, due to the diverse sources of data formats, this topic cannot be fixed at the beginning of workflow creation. Therefore, only after upstream operators perform data analysis can a specific topic be determined as the actual recorded image topic and passed to downstream operators as an environment variable. Transitive environment variables are obtained by retrieving their content from the .env environment variable file in the input path or shared path.

[0113] The priority of environment variables is: transitive > dynamic > static. When conflicts exist, the environment variable with higher priority should be selected.

[0114] Static environment variables are used to set the long-term configuration of operators, dynamic environment variables are provided when the workflow is triggered to specify specific runtime parameters, and transitive environment variables are dynamically generated during workflow execution to pass the output of the previous operator to subsequent operators. This hierarchical environment variable configuration mechanism ensures that data closed-loop workflows in complex application scenarios such as autonomous driving can operate flexibly and efficiently, while maintaining the accuracy and consistency of the data processing flow.

[0115] In one optional implementation, the corresponding operator container is invoked based on the execution order of each execution step in the workflow to obtain a task execution result matching the data processing task, further comprising:

[0116] S1, when there is at least one target operator of a target type among multiple operators, determine the target execution steps corresponding to each of the at least one target operator;

[0117] S2, when the target execution step is reached, the monitoring interface is called to query the operation status of the target operation;

[0118] S3, when the operation status is complete, calls the execution step after the target execution step.

[0119] In steps S1-S3 above, the target operator of the above target type can be a human-in-the-loop operator. Human-in-the-loop operators require human intervention. When the workflow executes a step related to this type of operator, the workflow engine will call a predefined monitoring interface to query the execution status of the target operator, ensuring that subsequent operator execution will not be accidentally triggered before the manual operation is completed.

[0120] In actual operation, many workflows require human intervention. For example, some processes require process approval before they can continue. In the data annotation process, after automated annotation, manual quality inspection and confirmation of the annotation results are required before downstream warehousing operations can proceed.

[0121] Compared to fully automated processing, when manual intervention is required, this operator registers the status result query service with the business management center when processing reaches a state that necessitates manual intervention. For example... Figure 8 As shown, pod2 is a human-in-the-loop operator. After receiving the request, the business management center can remind humans to participate in the business in two ways: push the actual business area of ​​the task to the human to participate; or the business area can periodically query the business management center to see if there are any new tasks.

[0122] While humans are actively involved in the work, the pod continuously polls the business management center via API calls to check the completion status of manual operations. It alternates between polling and sleeping until the manual operation status is returned. Based on the returned result, it may trigger the next operator in the workflow or terminate the process early if the requirements are not met.

[0123] The above process is described below using a complete implementation method:

[0124] like Figure 9 As shown, in the closed-loop processing of autonomous driving data, a workflow including automated labeling (DINO), manual quality inspection, and dataset entry is designed. In the design phase, the quality inspection operators are marked through configuration files to determine their positions in the workflow.

[0125] When the workflow reaches this step, it calls the monitoring interface of the business management center to query the current status of the quality inspection operator. If the status is pending, the workflow execution will be paused, waiting for the manual quality inspection to be completed.

[0126] Once the monitoring interface returns an approved status, the workflow engine will continue to execute the subsequent data entry steps, start the data entry operator container, store the quality-checked data into the database, and complete the entire data processing loop.

[0127] Through the above implementation methods, the workflow engine can effectively manage data pipelines that include complex operations such as human-in-the-loop, ensuring that each link in the data processing is properly monitored and controlled, thereby supporting higher levels of data processing automation and intelligence in autonomous driving scenarios and improving data processing efficiency.

[0128] In this application, the overall workflow from creation to completion is as follows:

[0129] Workflow Operator Selection Phase: Based on the data processing requirements, select the relevant operators that the workflow depends on. Each operator is a modular data processing module, representing a step in the overall data processing flow. This modular approach allows for better breakdown of complex processes into modules for sequential execution, and also helps accumulate generalized business units for quickly building new business processes. If human interaction is required during workflow execution, a human-in-the-loop operator must be added.

[0130] Workflow operation mode selection stage: After selecting the corresponding operator list, the operation mode of the entire workflow needs to be selected according to the operation mode of each operator in the operator list.

[0131] Specifically, this includes: when the selected operator list contains only serial mode operators, the operation is based on serial mode; when the selected operator list contains only shared mode operators, the operation is based on shared mode; when the selected operator list contains both serial and shared mode operators, the operation is based on mixed mode.

[0132] Workflow Configuration Phase: Following the workflow, each operator is linked together as a node to form a complete processing flow, i.e., the data processing workflow. The workflow is a Directed Acyclic Graph (DAG), meaning data processing is unidirectional. Nodes are connected in series, indicating the direction of processing. The workflow operation also requires a large number of environment variables, including basic environment configurations, required hardware, operating system environment, the actual data format, type, storage method, storage path, parameters and commands that operators depend on for deployment and execution, and the actual data processed, including message body names and types, depending on upstream and downstream relationships. During this execution phase, each operator is configured layer by layer for the various environment variables required in this workflow and operation, ensuring correct deployment and execution of the operators.

[0133] If the operator has static environment variables, they can be configured at this stage; if it includes human-in-the-loop operators, they can be orchestrated to the appropriate location so that when the subsequent workflow reaches the operator, it will pause execution and wait for human intervention.

[0134] Workflow Triggering Phase: In this phase, the relevant business parties actively trigger the corresponding workflow by calling the workflow service interface. Simultaneously, the workflow will also aggregate the dynamic environment variables required by all its included operators. When triggering the workflow, the relevant business parties must uniformly pass the corresponding parameter list; if an operator has dynamic environment variables, these can be obtained in this phase.

[0135] Workflow Execution Phase: In this phase, the workflow engine instantiates operators into pods sequentially according to the previously configured workflow, data transmission method, environment variables, and deployment method, and deploys them to Kubernetes. Simultaneously, it tracks the operational status of each operator. During the execution of each pod, the Kubernetes scheduler, considering the actual resource scheduling of the computing cluster and the operator's hardware and resource requirements, deploys the pod to the designated computing server for execution, monitoring its success, failure, and retry statuses to ensure the orderly and smooth execution of each module. For different operator types, whether fully automated or requiring manual intervention, the collaboration between the operator and the configuration center ensures that operators can still execute smoothly and orderly even with human intervention present.

[0136] The human-in-the-loop operator will block the automatic operation mechanism of the workflow at this stage until human intervention is performed to determine the subsequent execution plan of the workflow; if the operator has transitive environment variables, they can be obtained at this stage.

[0137] Workflow completion phase: Periodically clean up / destroy the operator's runtime resources.

[0138] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0139] According to another aspect of the embodiments of this application, a data task processing apparatus for implementing the above-described data task processing method is also provided. For example... Figure 10 As shown, the device includes:

[0140] The first determining unit 1002 determines the operators that match each execution step in the workflow of the data processing task based on the task description information of the data processing task, wherein the task description information is used to indicate the data processing logic of the data processing task.

[0141] The second determining unit 1004 determines the workflow operation mode matching the workflow based on the operator operation mode corresponding to each operator matched with each execution step. The operator operation mode is used to indicate the way the operator accesses data when performing data processing tasks.

[0142] Instantiation unit 1006 instantiates operators that match each execution step according to the workflow operation mode, and obtains operator containers that match each of the multiple operators. The operator containers include the execution code and environment variables corresponding to the execution steps.

[0143] The processing unit 1008 calls the corresponding operator container based on the execution order of each execution step in the workflow to obtain the task execution result that matches the data processing task.

[0144] Optionally, the second determining unit 1004 further includes: a third determining module, used to determine the current storage directory corresponding to the current input data set of the operator under the current execution step; and to determine the operator running mode of the operator under the current execution step according to the relationship between the directory corresponding to the first output data set of the operator under the first reference execution step and the current storage directory, wherein the first reference execution step is the previous execution step of the current execution step.

[0145] Optionally, the third determining module mentioned above includes: a fourth determining module, used to determine, when the current storage directory is the directory corresponding to the first output data set, that the operator running mode of the operator matching the current execution step is serial mode; and when the current storage directory is the directory corresponding to the first output data set and the second output data set, and the current input data set includes the first output data set and the second output data set, to determine that the operator running mode of the operator matching the current execution step is shared mode, wherein the second output data set is the output data set of the operator under the second reference execution step, and the second reference execution step is the execution step before the first reference execution step.

[0146] Optionally, the third determining module mentioned above includes: a fifth determining module, used to determine the workflow operation mode as a serial operation mode when the operator operation mode of the operator in each execution step in the workflow is a serial mode; to determine the workflow operation mode as a shared operation mode when the operator operation mode of the operator in each execution step in the workflow is a shared mode; and to determine the workflow operation mode as a mixed operation mode when the workflow contains operators in both serial and shared modes.

[0147] Optionally, the fifth determining module mentioned above includes: a sixth determining module, used to determine the input path and output path that match the multiple operators in the workflow when the workflow operation mode is a serial operation mode, wherein the operator obtains data in the first storage directory based on the input path, and the operator writes the data to the second storage directory based on the output path; configure read-only permissions for the first storage directory corresponding to the input path in the environment variables corresponding to the multiple operators, and configure read-write permissions for the second storage directory corresponding to the output path in the environment variables; and instantiate the multiple operators based on the environment variables corresponding to the multiple operators to obtain the operator container that matches the multiple operators.

[0148] Optionally, the fifth determining module mentioned above also includes a seventh determining module, used to determine the shared path matching all operators in the workflow when the workflow operation mode is a shared operation mode, wherein the operator obtains data in the shared storage directory based on the shared path, and the operator writes the data to the shared storage directory based on the shared path; configure read and write permissions for the shared storage directory corresponding to the shared path in the environment variables of each operator; and instantiate based on the environment variables of each operator to obtain the operator container matching each operator.

[0149] Optionally, the fifth determining module mentioned above further includes an eighth determining module, used to configure read-only permissions for the first storage directory corresponding to the input path in the environment variables of the serial mode operator when the workflow operation mode is a hybrid operation mode, configure read-write permissions for the second storage directory corresponding to the output path in the environment variables, and configure read-write permissions for the shared storage directory corresponding to the shared path in the environment variables of the shared mode operator; add a copy operator between the first reference operator in the serial mode and the second reference operator in the shared mode, which have an adjacent position relationship, wherein the copy operator is used to copy the data in the second storage directory corresponding to the output path of the first reference operator to the shared storage directory corresponding to the shared path of the second reference operator; and instantiate based on the environment variables of each operator and copy operator in the workflow to obtain an operator container that matches each operator and copy operator in the workflow.

[0150] Optionally, the first determining unit 1002 includes a configuration module, configured to configure static environment variables for operators of each execution step, wherein the static environment variables are used to indicate the type of configuration parameters; when a workflow is triggered, sequentially obtain the parameter data information corresponding to each execution step; configure dynamic environment variables for operators corresponding to each execution step based on the parameter data information, wherein the dynamic environment variables are used to indicate the type of sent parameters; when calling the corresponding operator container based on the execution order of each execution step in the workflow, configure transitive environment variables for the operator corresponding to the second execution step based on the execution result obtained from calling the operator container corresponding to the first execution step, wherein the execution order of the first execution step is before the second execution step, the priority of the transitive environment variable is greater than the priority of the dynamic environment variable, and the priority of the dynamic environment variable is greater than the priority of the static environment variable.

[0151] Optionally, the processing unit 1008 is further configured to, when there is at least one target operator of a target type among the multiple operators, determine the target execution steps corresponding to at least one target operator respectively; when the target execution step is executed, call the monitoring interface to query the operation status of the target operation; and when the operation status is a completed state, call the execution steps after the target execution step.

[0152] For specific implementation examples, please refer to the examples shown in the data task processing method described above, which will not be repeated here.

[0153] The memory 1102 can be used to store software programs and modules, such as the program instructions / modules corresponding to the data task processing method and apparatus in this embodiment of the invention. The processor 1104 executes various functional applications and data processing by running the software programs and modules stored in the memory 1102, thereby realizing the aforementioned data task processing method. The memory 1102 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1102 may further include memory remotely located relative to the processor 1104, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Specifically, the memory 1102 may be used, but is not limited to, for storing file information such as target files. As an example, such as Figure 11 As shown, the memory 1102 may include, but is not limited to, the first determining unit 1002, the second determining unit 1004, the instantiation unit 1006, and the processing unit 1008 in the data task processing device. Furthermore, it may include, but is not limited to, other module units in the data task processing device, which will not be described further in this example.

[0154] Optionally, the aforementioned transmission device 1106 is used to receive or send data via a network. Specific examples of the network may include wired and wireless networks. In one example, the transmission device 1106 includes a Network Interface Controller (NIC), which can be connected to other network devices and a router via a network cable to communicate with the Internet or a local area network. In another example, the transmission device 1106 is a radio frequency (RF) module used for wireless communication with the Internet.

[0155] In addition, the above-mentioned electronic device also includes a display 1108 and a connection bus 1110 for connecting the various module components in the above-mentioned electronic device.

[0156] According to one aspect of this application, a computer program product is provided, comprising a computer program / instructions containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by a central processing unit, it performs various functions provided in embodiments of this application.

[0157] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0158] It should be noted that the computer system of the electronic device is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0159] Specifically, according to embodiments of this application, the processes described in the various method flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by a central processing unit, it performs various functions defined in the system of this application.

[0160] According to one aspect of this application, a computer-readable storage medium is provided, wherein a processor of a computer device reads computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the methods provided in the various alternative implementations described above.

[0161] Optionally, in this embodiment, the computer-readable storage medium described above may be configured to store a computer program for performing the following steps:

[0162] S1, Based on the task description information of the data processing task, determine the operators in the workflow of the data processing task that match each execution step, wherein the task description information is used to indicate the data processing logic of the data processing task.

[0163] S2, determine the workflow operation mode that matches the workflow based on the operator operation mode corresponding to each operator that matches each execution step, wherein the operator operation mode is used to indicate the way the operator accesses data when performing data processing tasks;

[0164] S3, instantiate the operators that match each execution step according to the workflow operation mode to obtain the operator container that matches each of the multiple operators. The operator container includes the execution code and environment variables corresponding to the execution step.

[0165] S4 calls the corresponding operator container based on the execution order of each execution step in the workflow to obtain the task execution result that matches the data processing task.

[0166] Optionally, in this embodiment, those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware of an electronic device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0167] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0168] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods of the various embodiments of this application.

[0169] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0170] In the several embodiments provided in this application, it should be understood that the disclosed user equipment can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of units or modules may be electrical or other forms.

[0171] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0172] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0173] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for processing data tasks, characterized in that, include: Based on the task description information of the data processing task, determine the operators in the workflow of the data processing task that match each execution step, wherein the task description information is used to indicate the data processing logic of the data processing task; Based on the operator operation mode corresponding to each of the operators matched with each of the execution steps, a workflow operation mode matching the workflow is determined, wherein the operator operation mode is used to indicate the way the operator accesses data when executing the data processing task; According to the workflow operation mode, the operators that match each of the execution steps are instantiated to obtain operator containers that match each of the multiple operators. The operator containers include the execution code and environment variables corresponding to the execution steps. Based on the execution order of each execution step in the workflow, the corresponding operator container is invoked to obtain the task execution result that matches the data processing task.

2. The method according to claim 1, characterized in that, Before determining the workflow execution mode matching the workflow based on the operator execution mode corresponding to each of the operators matching the respective execution steps, the process includes: Determine the current storage directory corresponding to the current input data set of the operator in the current execution step; Based on the relationship between the directory corresponding to the first output data set of the operator under the first reference execution step and the current storage directory, the operator operation mode of the operator in the current execution step is determined, wherein the first reference execution step is the previous execution step of the current execution step.

3. The method according to claim 2, characterized in that, The step of determining the operator running mode of the operator in the current execution step based on the relationship between the directory corresponding to the first output data set of the operator in the first reference execution step and the current storage directory includes: If the current storage directory is the directory corresponding to the first output data set, the operator running mode of the operator matching the current execution step is determined to be serial mode; When the current storage directory is the directory corresponding to the first output data set and the second output data set, and the current input data set includes the first output data set and the second output data set, the operator running mode of the operator matching the current execution step is determined to be a shared mode, wherein the second output data set is the output data set of the operator under the second reference execution step, and the second reference execution step is the execution step before the first reference execution step.

4. The method according to claim 3, characterized in that, The step of determining the workflow operation mode matching the workflow based on the operator operation mode corresponding to each of the operators matching the respective execution steps includes: If the operator running mode of the operator in each execution step of the workflow is the serial mode, then the workflow running mode is determined to be the serial running mode. If the operator running mode of the operator in each execution step of the workflow is the shared mode, then the workflow running mode is determined to be the shared running mode. When the workflow includes both the serial mode operator and the shared mode operator, the workflow operation mode is determined to be a hybrid operation mode.

5. The method according to claim 4, characterized in that, The step of instantiating operators matching each execution step according to the workflow operation mode to obtain operator containers matching each of the multiple operators includes: When the workflow operation mode is the serial operation mode, the input path and output path that match the multiple operators in the workflow are determined respectively, wherein the operator obtains data in the first storage directory based on the input path, and the operator writes the data to the second storage directory based on the output path; Configure read-only permissions for the first storage directory corresponding to the input path in the environment variables corresponding to each of the multiple operators, and configure read-write permissions for the second storage directory corresponding to the output path in the environment variables; Instantiate the environment variables corresponding to each of the multiple operators to obtain the operator container that matches each of the multiple operators.

6. The method according to claim 4, characterized in that, The step of instantiating operators matching each execution step according to the workflow operation mode to obtain operator containers matching each of the multiple operators includes: When the workflow operation mode is the shared operation mode, a shared path matching all the operators in the workflow is determined, wherein the operator obtains data in the shared storage directory based on the shared path, and the operator writes the data to the shared storage directory based on the shared path; Configure read and write permissions for the shared storage directory corresponding to the shared path in the environment variables of each operator; Instantiate based on the environment variables of each operator to obtain the operator container that matches each operator.

7. The method according to claim 4, characterized in that, The step of instantiating operators matching each execution step according to the workflow operation mode to obtain operator containers matching each of the multiple operators includes: When the workflow operation mode is the hybrid operation mode, read-only permissions are configured for the first storage directory corresponding to the input path in the environment variables of the operator in the serial mode, read-write permissions are configured for the second storage directory corresponding to the output path in the environment variables, and read-write permissions are configured for the shared storage directory corresponding to the shared path in the environment variables of the operator in the shared mode. A copy operator is added between the first reference operator of the serial mode and the second reference operator of the shared mode, which have an adjacent position relationship. The copy operator is used to copy the data in the second storage directory corresponding to the output path of the first reference operator to the shared storage directory corresponding to the shared path of the second reference operator. Instantiate the environment variables of each operator and copy operator in the workflow to obtain the operator container that matches each operator and copy operator in the workflow.

8. The method according to claim 1, characterized in that, After determining the operators matching each execution step in the workflow of the data processing task based on the task description information of the data processing task, the process includes: Static environment variables are configured for the operators of each of the execution steps, wherein the static environment variables are used to indicate the type of the configuration parameters; When the workflow is triggered, the parameter data information corresponding to each of the execution steps is obtained sequentially. Based on the transmitted parameter data information, dynamic environment variables are configured for the operators corresponding to each of the execution steps, wherein the dynamic environment variables are used to indicate the type of transmitted parameters; When the corresponding operator container is invoked based on the execution order of each execution step in the workflow, a transitive environment variable is configured for the operator corresponding to the second execution step based on the execution result obtained by invoking the operator container corresponding to the first execution step. The execution order of the first execution step is before the second execution step, the priority of the transitive environment variable is greater than the priority of the dynamic environment variable, and the priority of the dynamic environment variable is greater than the priority of the static environment variable.

9. The method according to claim 1, characterized in that, The step of calling the corresponding operator container based on the execution order of each execution step in the workflow to obtain a task execution result matching the data processing task further includes: In the case where there is at least one target operator of a target type among the multiple operators, determine the target execution steps corresponding to each of the at least one target operator; When the target execution step is reached, the monitoring interface is called to query the operation status of the target operation; If the operation status is complete, the execution step following the target execution step is invoked.

10. A data task processing apparatus, characterized in that, include: The first determining unit determines the operators that match each execution step in the workflow of the data processing task based on the task description information of the data processing task, wherein the task description information is used to indicate the data processing logic of the data processing task. The second determining unit determines a workflow operation mode that matches the workflow based on the operator operation mode corresponding to each of the operators that match each of the execution steps, wherein the operator operation mode is used to indicate the way the operator accesses data when executing the data processing task; An instantiation unit instantiates operators that match each of the execution steps according to the workflow operation mode, to obtain operator containers that match each of the multiple operators, wherein the operator containers include execution code and environment variables corresponding to the execution steps; The processing unit calls the corresponding operator container based on the execution order of each execution step in the workflow to obtain the task execution result that matches the data processing task.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program is executed by an electronic device to perform the method according to any one of claims 1 to 9.

12. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to execute the method described in any one of claims 1 to 9 through the computer program.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

Images