Task dynamic parameter transmission method and device based on big data execution engine

Abstract

The invention particularly relates to a task dynamic parameter transmission method and device based on a big data execution engine, and relates to the technical field of big data processing, and the method comprises the steps: according to the type of an upstream task, adopting a parameter processing logic adaptive to the type of the task, and storing three types of parameters to a corresponding storage medium; two preconditions are verified, wherein the two preconditions include the dependency relationship between the downstream task configured in the platform and the upstream task, the upstream task completing output parameter configuration and the parameter state being available. According to the method, three types of parameters of constant, user-defined operation and calculation results are divided, different business scenes such as static configuration transmission, cross-task unified management and dynamic result reuse are accurately matched, and diversified parameter transmission requirements such as database connection ports, date partitions and task execution results are covered; specific parameter output logic is designed for four mainstream task types of SparkSQL, Python, Shell and RDB, and a standardized storage path and a cross-task multiplexing mechanism are adopted, so that the problems of repeated configuration and format incompatibility are avoided.

Description

Technical Field

[0001] This invention relates to the field of big data processing technology, and in particular to a method and apparatus for dynamic parameter passing of tasks based on a big data execution engine. Background Technology

[0002] In the field of big data task scheduling and processing, to achieve dynamic parameter transfer between tasks, existing technologies offer several typical solutions, all of which aim to solve the problem of data or instructions flowing between upstream and downstream tasks: A metadata-based key-value storage scheme: Upstream tasks write output parameters as key-value pairs to a public metadata database (such as MySQL or Redis). Downstream tasks retrieve the corresponding values ​​from this database by querying a predefined key. Asynchronous delivery based on message queues: Upstream tasks publish parameters as messages to message topics (such as topics in Kafka and RabbitMQ), and downstream tasks subscribe to the topic to consume messages; Service-based retrieval based on API / configuration center: Parameters are hosted in a configuration center or parameter service (such as Nacos, Consul, or internal configuration platform), and downstream applications retrieve the latest parameters via API before or during runtime.

[0003] In practical applications, especially when dealing with complex and heterogeneous big data workflows, the aforementioned existing technical solutions have revealed the following shortcomings that urgently need to be addressed or improved: Lack of uniformity and platform support: Different task types (such as SparkSQL, Shell, RDB) may require different parameter passing methods, and there is a lack of a unified platform-level interface for declaration and management. This leads to a fragmented technology stack, inconsistent user experience, and increased complexity in development and operation.

[0004] Limited parameter passing capability makes it difficult to reuse "calculation results": Existing solutions (especially those based on standard output or simple key-value storage) are usually only suitable for passing a small amount of metadata or simple scalar values. They are difficult to efficiently and structurally pass a complete result set of an SQL query (such as table-level statistical data, multi-row and multi-column data) as parameters, and cannot truly achieve the reuse of complex "calculation results". Decoupled from task dependencies, resulting in poor observability: The parameter passing chain is often separated from the dependencies managed by the task scheduler. The platform cannot intuitively show how parameters flow with task dependencies, and it is also difficult to effectively monitor and manage the success or failure of parameter passing and the correctness of parameter values, posing risks to data consistency and timeliness. Introducing additional dependencies and performance bottlenecks: Solutions based on shared databases or message queues introduce external system dependencies, the availability and performance of which directly impact the stability of the entire workflow. Furthermore, frequent small file I / O or database queries can also become performance bottlenecks.

[0005] Therefore, a method and apparatus for dynamic parameter passing of tasks based on a big data execution engine is proposed to address the aforementioned problems. Summary of the Invention

[0006] The purpose of this invention is to provide a method and apparatus for dynamic parameter passing of tasks based on a big data execution engine in order to solve the above problems.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: A method for dynamic parameter passing in tasks based on a big data execution engine includes: Define three types of parameters: constant parameters, user-defined runtime parameters, and calculation result parameters; Based on the type of upstream task, parameter processing logic adapted to the task type is adopted to store the three types of parameters in the corresponding storage medium; Verify two prerequisites: the downstream task has been configured with the dependency relationship with the upstream task in the platform, and the upstream task has completed the configuration of output parameters and the parameter status is available. Based on the type of the upstream output parameters, the parameter values ​​are retrieved from the corresponding storage medium and injected into the execution context of the downstream task using a referencing method that is compatible with the parameter type.

[0008] Preferably, the definition of the three types of parameters—constant parameters, user-defined running parameters, and calculation result parameters—specifically includes: Constant parameters: These are fixed character or numeric data used to transmit static configurations that do not require dynamic calculation. Custom runtime parameters: Define parameters in a format that supports referencing platform-built-in variables or user-defined variables, enabling unified management of parameters for upstream and downstream tasks; Calculation result parameters: Parameters dynamically generated based on task execution results, obtained by writing specific calculation logic.

[0009] Preferably, the types of upstream tasks include SparkSQL tasks, Python tasks, Shell tasks, and RDB tasks; SparkSQL task output parameters include: Before submitting tasks to the Spark cluster, the data platform performs format conversion on the output parameter statements and supplements the ORC storage format declaration and output path configuration. After receiving the task, the SparkSQL execution engine extracts the core query statement by matching the format using regular expressions, and writes the query results to the specified path in HDFS in ORC format; the platform will trigger automatic verification logic after the task is completed.

[0010] Preferably, the Python task output parameters are implemented as follows: Before submitting a Python task, the platform embeds the user-written output parameter statements into a pre-defined standardized script template. Dynamic configuration of environment parameters: The platform will add dedicated environment variables for each task to specify the name and storage path of the output file. The specific parameter configurations are as follows: The output file name should follow the same format as the SparkSQL task name; the parent directory of the output file; After the Python task is completed locally, the built-in file upload client is invoked to upload the locally generated parameter file to the corresponding HDFS directory based on the path information in the environment variables.

[0011] Preferably, the Shell task output parameters are implemented as follows: Before submitting the Shell task, the platform combines the user-configured output parameter statements with the pre-generated output path to form a complete writing script, which is then appended to the end of the user's original execution script; the output path format is consistent with that of the SparkSQL task. The Shell execution engine directly runs the concatenated script and writes the v1 string to the specified local path on the Linux server. If the local path does not exist, the platform will create it automatically; if an error occurs during the writing process, the task will terminate immediately and return an error code. If a downstream task needs to read this parameter file across servers, the platform will automatically synchronize the local file to shared storage or upload it directly to HDFS after the Shell task is executed.

[0012] Preferably, the RDB task output parameters are implemented as follows: Before submitting the RDB task, the SQL parsing engine is called to perform syntax validation and semantic analysis on the SQL statement corresponding to the output parameters; The platform encapsulates the parsed SQL statement into a standardized execution request structure and submits it to the RDB execution engine. The engine first executes the core business script written by the user. After the script executes successfully, it supplements the SQL statement corresponding to the execution parameters and obtains the query result set. Since the result set of an RDB task is a two-dimensional table structure, the platform will convert it into a key-value pair format of row index-column name-value, and store it in the platform's metadata database as a JSON string, while also recording the upstream metadata corresponding to the parameters.

[0013] Preferably, the step of retrieving parameter values ​​from the corresponding storage medium and injecting them into the execution context of the downstream task using a referencing method adapted to the parameter type, based on the type of the upstream output parameters, specifically includes: Obtaining constants and custom runtime parameters: Downstream tasks declare the upstream parameters they reference in the platform configuration interface, and the platform automatically records the relationships. Before the downstream task is executed, the platform calls the metadata database query interface to query the latest parsed values ​​of the upstream parameters based on the correlation; then the parameter values ​​are injected into the execution context of the downstream task in the form of environment variables or script variables. Before executing downstream tasks, the platform re-verifies the status of upstream parameters. If a parameter is missing or expired, the platform immediately terminates the execution of the downstream task and sends an alarm.

[0014] Preferably, the process further includes obtaining the calculation result parameters: If the upstream output is a calculation result parameter, since the storage location of such parameters varies depending on the upstream task type, the downstream task adopts a targeted reading strategy based on the upstream task engine type: Upstream parameter retrieval for SparkSQL / Python tasks: The computation results parameters of both SparkSQL and Python tasks are stored in HDFS as files. Downstream tasks obtain parameter values ​​by reading HDFS files and parsing them in a structured manner. Two reading modes are supported: row-column combined reading mode and columnar reading mode. Upstream parameter retrieval for Shell tasks: The computation results of Shell tasks are stored by default in the local path of the execution node. Downstream tasks select the corresponding reading method based on the file storage location, including: reading from the same node and reading from across nodes; upstream tasks retrieve parameters for RDB tasks. The computation results of RDB tasks are stored in the platform's metadata database in JSON format. Downstream tasks obtain these results through JSON parsing and structured references, including parameter querying and JSON parsing, structured references, and batch data processing.

[0015] Preferably, the method further includes: Through parameter version management, end-to-end monitoring, and three-level access control mechanisms, the traceability, reliability, and data security of parameter transmission are ensured. Specifically, this includes: The platform generates a unique version number for each output parameter; The platform monitors the entire link status of parameters from upstream generation to downstream reference in real time; the platform controls access to parameters based on a three-level permission system of tenant, project, and task.

[0016] A task dynamic parameter passing device based on a big data execution engine, comprising: The parameter classification definition module is configured to be divided into three categories: explicit constant parameters, custom running parameters, and calculation result parameters, defining their data characteristics, configuration methods, and core uses respectively. The upstream task parameter output module is configured to handle four types of upstream tasks: SparkSQL, Python, Shell, and RDB. It uses processing logic adapted to the task type to store parameters in HDFS, local storage, or a metadata database. The downstream task parameter reference acquisition module is configured to first verify the upstream and downstream task dependency configurations and the availability status of upstream parameters, and then obtain the parameter values ​​according to the upstream parameter type and upstream task type and inject them into the downstream task execution context. The parameter transfer guarantee mechanism module is configured to ensure the traceability, reliability, and data security of parameter transfer through parameter version management, end-to-end status monitoring, and access control.

[0017] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: 1. This invention precisely matches different business scenarios such as static configuration transmission, unified cross-task management, and dynamic result reuse by dividing parameters into three categories: constants, custom execution, and calculation results. It covers diverse parameter transmission needs such as database connection ports, date partitions, and task execution results. For four mainstream task types, SparkSQL, Python, Shell, and RDB, it designs dedicated parameter output logic and adopts standardized storage paths and cross-task reuse mechanisms to avoid duplicate configurations and format incompatibility issues.

[0018] 2. This invention supports diverse modes such as row-column combination, columnar reading, and batch injection through downstream referencing, adapting to different use cases such as precise value retrieval and SQLIN condition concatenation. At the same time, it simplifies the operation process through mechanisms such as automatic parameter injection and cross-node file synchronization. The exception handling mechanisms such as file size verification, upload retry, and SQL syntax verification further reduce the risk of parameter generation and referencing failure, greatly improve the overall efficiency of big data task flow, and reduce the cost of manual intervention. Attached Figure Description

[0019] Further details, features, and advantages of this application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which: Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a system module diagram of the present invention. Detailed Implementation

[0020] Several embodiments of this application will now be described in more detail with reference to the accompanying drawings to enable those skilled in the art to implement this application. This application may be embodied in many different forms and for various purposes and should not be limited to the embodiments set forth herein. These embodiments are provided to make this application thorough and complete, and to fully convey the scope of this application to those skilled in the art. The embodiments described do not limit this application.

[0021] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It will be further understood that terms such as those defined in commonly used dictionaries shall be interpreted as having a meaning consistent with their meaning in the relevant field and / or the context of this specification, and shall not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0022] Example 1 Its specific implementation method is combined with the appendix Figure 1 and attached Figure 2 Please provide a detailed explanation.

[0023] Appendix Figure 1 The flowchart of a task dynamic parameter passing method based on a big data execution engine provided in this embodiment of the invention shows the complete steps from defining three types of parameters to obtaining parameter values ​​from the corresponding storage medium and injecting them into the execution context of the downstream task according to the type of upstream output parameters and the reference method adapted to the parameter type.

[0024] Appendix Figure 2 This invention provides a structural block diagram of a task dynamic parameter passing method based on a big data execution engine, which shows the connection relationship between the parameter classification definition module and the parameter flow guarantee mechanism module, and marks the main functional interaction flow of each module.

[0025] In this embodiment, it includes: Upstream task output parameter generation: Define three types of parameters: constant parameters, user-defined runtime parameters, and calculation result parameters; Specifically, it includes: Constant parameters: These are fixed character or numerical data such as "abc" or "12323". They are mainly used to pass static configurations that do not require dynamic calculation (such as database connection ports, fixed filtering conditions, etc.). When configuring, you can directly enter the specific value in the platform interface to complete the definition.

[0026] Custom runtime parameters: Defined in key-value pairs, supporting the use of platform-built-in variables (such as $[yyyy-MM-dd] representing the current date and $[project_id] representing the project identifier) ​​or user-defined variables. The core function is to achieve unified management of parameters for upstream and downstream tasks (such as multiple tasks sharing the same date partition parameter), avoiding the risks of duplicate configuration and inconsistency.

[0027] Calculation result parameters: These are parameters that are dynamically generated based on the task execution results and need to be obtained by writing specific calculation logic. For example, a SparkSQL task queries table data using "select a,b from table1", a Shell task outputs characters using "echo a", a Python task prints the result using "print 'testdata'", and a MySQL task queries database data using "select x from table", ultimately outputting the calculation result as a parameter.

[0028] Based on the type of upstream task, parameter processing logic adapted to the task type is adopted to store the three types of parameters in the corresponding storage media (including HDFS, local storage, and metadata database), and the storage path follows a standardized format. The types of upstream tasks include SparkSQL tasks, Python tasks, Shell tasks, and RDB tasks; SparkSQL task output parameters include: Pre-submission transformation: Before submitting a task to the Spark cluster, the data platform automatically transforms the output parameter statements, supplementing the ORC storage format declaration and output path configuration. An example of the transformed statement is shown below: output directory '${path}' stored as orc select a, b from table1; Wherein, ${path} is a standardized file path pre-generated by the platform, in the format ${tenant id} / ${project identifier} / output_${parameter name}_${task id}_${scheduled time}; the tenant id and project identifier are used to achieve resource isolation, the parameter name and task id are used to uniquely identify the parameter source, and the scheduled time (e.g., 202510081200) is used to distinguish parameters for different scheduling cycles and avoid file overwriting.

[0029] Post-task execution storage: After receiving the task, the Spark SQL execution engine uses regular expressions to match the format "outputdirectory'dir_path'select_statement" to extract the core query statement (select a, b from table1); then, using the Spark DataFrame API, it writes the query results to the specified path in HDFS in ORC format (high compression ratio, suitable for columnar queries). File size verification and exception handling: To avoid downstream task read performance degradation or storage resource consumption exceeding limits due to excessively large parameter files, the platform will trigger automatic verification logic after the task execution is completed; by querying the total file size under the corresponding path through HDFSAPI, if it exceeds the preset threshold (default 2MB, supports user-defined adjustment), the task will be immediately marked as "parameter generation failed", and the cleanup interface will be called to delete the generated temporary files. At the same time, an alarm notification will be sent to the task manager via SMS and email.

[0030] Python task output parameter implementation: If the upstream task type is Python and the output parameter is configured as "print 'testdata'" (calculation result parameter), the platform will implement parameter output through a dual mechanism of script injection and environment parameter configuration. The specific process is as follows: Automatic output script injection: Before submitting a Python task, the platform will embed the user-written output parameter statement (print 'testdata') into a preset standardized script template, supplementing the logic for file reading and writing, standard output redirection, etc. Dynamic configuration of environment parameters: The platform will add dedicated environment variables for each task to specify the name and storage path of the output file. The specific parameter configurations are as follows: OUTPUT_FILENAME: The filename of the task output file, in the format output_${parameter name}_${task id}_${scheduled time}, and should be consistent with the naming of SparkSQL tasks; OUTPUT_PARENT_DIR: The parent directory of the output files, in the format ${tenant id} / ${project identifier}, to ensure file isolation between different tenants and projects; In addition, the platform declares a list of output files (supporting multiple file outputs) through the internal.application.outputfiles parameter and synchronizes the HDFS parent directory path through the internal.application.output.parentdir parameter for subsequent file uploads.

[0031] Upload the execution result to HDFS: After the Python task is completed locally, the platform will call the built-in file upload client to upload the locally generated parameter file (such as output_param1_task001_202510081200) to the corresponding HDFS directory according to the path information in the environment variables, and verify the integrity of the file (by comparing the MD5 value). If the upload fails, it will be retried 3 times. If it still fails, the task will be marked as abnormal.

[0032] Shell task output parameter implementation: When the upstream task type is Shell and the output parameter is configured as "echov1" (calculation result parameter), the platform adopts a simplified processing method of script concatenation and direct writing, as follows: Output script concatenation: Before submitting the Shell task, the platform combines the user-configured output parameter statement (echov1) with the pre-generated output path to concatenate a complete writing script, which is then appended to the end of the user's original execution script. The output path format is completely consistent with SparkSQL tasks, ensuring consistent path rules across task types. Local storage of execution results: The shell execution engine (such as bash) directly runs the concatenated script and writes the v1 string to the specified local path on the Linux server (such as / data / tenant001 / project001 / output_param2_task002_202510081200). If the local path does not exist, the platform will automatically create it (using the `mkdir -p` command); if an error such as insufficient permissions occurs during the writing process, the task will terminate immediately and return an error code (such as 13). Subsequent synchronization mechanism: If downstream tasks need to read the parameter file across servers, the platform will automatically synchronize the local file to shared storage (such as NFS) or directly upload it to HDFS after the Shell task is executed (reusing the upload logic of Python tasks). The specific synchronization method can be selected through the task configuration interface.

[0033] RDB task output parameter implementation: If the upstream task type is RDB (such as MySQL or PostgreSQL) and the output parameter is configured as "selectxfromtable" (calculation result parameter), the platform outputs the parameter through SQL parsing, result encapsulation, and JSON storage, which differs from the file storage method of other tasks. The specific process is as follows: SQL statement parsing and validation: Before submitting an RDB task, the platform will call an SQL parsing engine (such as Antlr) to perform syntax validation and semantic analysis on the SQL statement (selectxfromtable) corresponding to the output parameters to ensure that the statement is valid (such as no syntax errors and the existence of query fields). If the parsing fails, a configuration error message will be returned directly. Task execution and result acquisition: The platform encapsulates the parsed SQL statement into a standardized execution request structure (SingleSqlInfo) and submits it to the RDB execution engine. The engine first executes the core business script written by the user (such as data cleaning and update operations). After the script is executed successfully, the SQL statement corresponding to the execution parameters is added to obtain the query result set (two-dimensional table structure, such as multiple rows of data containing the x field). Result JSON storage: Since the result set of an RDB task is a two-dimensional table structure, the platform will convert it into a key-value pair format of row index-column name-value (e.g., {"0":{"x":"value1"},"1":{"x":"value2"}}) and store it in the platform's metadata database (e.g., PostgreSQL) as a JSON string. At the same time, it will record metadata such as the upstream task ID and generation time corresponding to the parameters, which will facilitate downstream task query and traceability.

[0034] Downstream task input parameter references: Verify two prerequisites: the downstream task has been configured with the dependency relationship with the upstream task in the platform, and the upstream task has completed the configuration of output parameters and the parameter status is available. Based on the type of upstream output parameters (constant parameters, custom running parameters, calculation result parameters), the parameter values ​​are retrieved from the corresponding storage medium and injected into the execution context of the downstream task using a referencing method that is compatible with the parameter type.

[0035] Specifically, it includes: Obtaining constants and custom runtime parameters: If the upstream output is a constant parameter or a custom runtime parameter, since these parameters are already stored in the platform metadata database in key-value pairs, downstream tasks can retrieve them directly by querying the parameter name, making the process simple and efficient. In the platform configuration interface, downstream tasks declare the upstream parameters to be referenced by selecting the upstream task and the parameter name (e.g., selecting the parameter "date_partition" of upstream task TaskA). The platform will automatically record the association relationship between downstream task ID, upstream task ID, and parameter name. Parameter value query and injection: Before the downstream task is executed, the platform calls the metadata database query interface to query the latest parsed value of the upstream parameter based on the relationship (such as the parsed value of the custom running parameter "date_partition" being "2025-10-08"); then injects the parameter value into the execution context of the downstream task in the form of environment variables or script variables; For example, in a SparkSQL task, ${date_partition} is referenced directly; in a Shell task, it is obtained through ${date_partition}; and in a Python task, it is read through ${date_partition}.

[0036] Parameter consistency verification: To avoid downstream reference anomalies caused by upstream parameter updates, the platform will re-verify the status of upstream parameters (must be "available" and not deleted) before the downstream task is executed. If the parameter does not exist or has expired, the execution of the downstream task will be terminated immediately and an alarm will be sent.

[0037] It also includes obtaining the parameters of the calculation results: If the upstream output is computation result parameters, the storage location of such parameters (HDFS, local file, metadata database) varies depending on the upstream task type. Therefore, the downstream task needs to adopt a targeted reading strategy based on the upstream task engine type. Upstream parameter retrieval for SparkSQL / Python tasks: The computation results parameters of both SparkSQL and Python tasks are stored in HDFS as files (ORC or text format). Downstream tasks obtain parameter values ​​by reading HDFS files and parsing them in a structured manner, supporting two reading modes: Row-column combined reading mode: suitable for scenarios that require precise acquisition of data in a specific row and column. Downstream tasks reference parameters in the format ${parameter name}[row index][column index] (e.g., ${param_from_spark}[0][1]), where both row and column indices are counted from 0. When the platform executes, it first reads the ORC / text file in HDFS, parses it into a two-dimensional array structure, and then extracts the corresponding data according to the index; for example, the ORC file output by the upstream SparkSQL task contains two columns (a, b) and three rows of data, and ${param_from_spark}[2][0] represents reading the value of the 3rd row and 1st column (column a); Columnar read mode: Suitable for scenarios that require retrieving all data from a specific column. Downstream tasks reference the column using the format ${parameter name}[*][column index] (e.g., ${param_from_spark}[*][1]). The platform will parse the file and return all data for that column (e.g., ["value1","value2","value3"]). It can be used directly as an IN condition in SQL queries (e.g., wherebin(${param_from_spark}[*][1])) or as a list variable in Python scripts.

[0038] Upstream parameter retrieval for Shell tasks: The calculation results of Shell tasks are stored by default in the local path of the execution node (or have been synchronized to shared storage). Downstream tasks need to select the corresponding reading method based on the file storage location. Same-node reading: If the downstream task and the upstream Shell task are executed on the same Linux server, the downstream task can read directly through the local file path; the platform will inject the local path of the upstream parameter (such as / data / tenant001 / project001 / output_param2_task002_202510081200) into the environment variables of the downstream task, and the downstream Shell task can read the file content through cat${SHELL_PARAM_PATH}; Cross-node reading: If the downstream task and the upstream Shell task are not on the same node, the platform will first synchronize the upstream local parameter file to HDFS (reusing the upload logic of the Python task), and the downstream task will then obtain the parameters through HDFS file reading, which is consistent with the parameter reading logic of SparkSQL / Python tasks.

[0039] Upstream parameter retrieval for RDB tasks: The computation results of the RDB task are stored in the platform's metadata database in JSON format, and downstream tasks retrieve them through JSON parsing and structured references. Parameter query and JSON parsing: After the downstream task configuration references the upstream RDB parameters, the platform will query the corresponding JSON string from the metadata database (such as {"0":{"x":"value1"},"1":{"x":"value2"}}), and convert it into operable structured data (such as Map objects in Java or dictionaries in Python) through a JSON parser (such as Jackson). Structured references: Downstream tasks support the same row and column combination reading mode as SparkSQL tasks, referencing parameters in the format ${parameter name}[row index][column name] (e.g., ${param_from_rdb}[1][x], which means reading the value "value2" in the second row and x column), without needing to worry about the underlying storage format of JSON, thus lowering the barrier to entry for users; Batch data processing: If the upstream RDB parameters contain a large amount of data (such as 1000 rows), the platform supports batch injection of parameter values ​​into downstream SQL tasks, such as generating filter conditions like wherexin('value1','value2',...,'value1000'), avoiding the tedious operation of manually concatenating SQL statements.

[0040] Parameter flow guarantee: Through parameter version management, end-to-end monitoring, and three-level access control mechanisms, the traceability, reliability, and data security of parameter transmission are ensured. Specifically, this includes: Parameter version management mechanism: The platform generates a unique version number for each output parameter (format: ${task id}_${parameter name}_${execution timestamp}), and records metadata such as the parameter's generation time, source task, and storage location. When the upstream task is re-executed to generate new parameters, the old version parameter will not be deleted. It supports downstream tasks to roll back and reference historical versions, avoiding business anomalies caused by parameter overwriting.

[0041] Parameter transmission monitoring mechanism: The platform monitors the entire link status of parameters from upstream generation to downstream reference in real time, including upstream parameter generation success / failure, parameter file upload success / failure, downstream parameter reading success / failure, etc. If an abnormality occurs at a certain node (such as parameter file upload timeout), the platform will immediately trigger an alarm and display detailed logs of the abnormal node (such as error code, exception stack), which facilitates quick problem location. Access control mechanism: The platform strictly controls access to parameters based on a three-level permission system of tenant, project, and task; only downstream tasks with viewing permissions for upstream tasks and configured dependencies can reference the corresponding parameters; cross-tenant and cross-task references to parameters are prohibited to ensure data security and resource isolation.

[0042] Example 2 Please see Figure 2 A task dynamic parameter passing device based on a big data execution engine includes the following parts: The parameter classification definition module is configured to be divided into three categories: explicit constant parameters, custom running parameters, and calculation result parameters, defining their data characteristics, configuration methods, and core uses (such as static configuration transfer, unified management, and dynamic calculation generation). The upstream task parameter output module is configured to handle four types of upstream tasks: SparkSQL, Python, Shell, and RDB. It uses processing logic adapted to the task type (such as format conversion, script injection, SQL parsing, etc.) to store parameters in HDFS, local storage, or metadata database, and follows standardized storage paths and exception handling rules. The downstream task parameter reference acquisition module is configured to first verify the upstream and downstream task dependency configurations and the availability status of upstream parameters, and then, according to the upstream parameter type (constant / custom execution / calculation result) and the upstream task type, use adaptation methods such as direct query, file reading, and JSON parsing to obtain the parameter values ​​and inject them into the downstream task execution context. The parameter transfer guarantee mechanism module is configured to ensure the traceability, reliability, and data security of parameter transfer through parameter version management (supporting historical rollback), full-link status monitoring (abnormal alarms), and three-level permission control of tenant-project-task.

[0043] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0044] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

[0045] It should be noted that, in this document, the use of relational terms such as "first" and "second" is merely for distinguishing one entity or operation from another, and does not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0046] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0047] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0048] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0049] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0050] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0051] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0052] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A method for dynamic parameter passing in tasks based on a big data execution engine, characterized in that, include: Define three types of parameters: constant parameters, user-defined runtime parameters, and calculation result parameters; Based on the type of upstream task, parameter processing logic adapted to the task type is adopted to store the three types of parameters in the corresponding storage medium; Verify two prerequisites: the downstream task has been configured with the dependency relationship with the upstream task in the platform, and the upstream task has completed the configuration of output parameters and the parameter status is available. Based on the type of the upstream output parameters, the parameter values ​​are retrieved from the corresponding storage medium and injected into the execution context of the downstream task using a referencing method that is compatible with the parameter type.

2. The method for dynamic parameter passing of tasks based on a big data execution engine according to claim 1, characterized in that, Define three types of parameters: constant parameters, user-defined runtime parameters, and calculation result parameters, specifically including: Constant parameters: These are fixed character or numeric data used to transmit static configurations that do not require dynamic calculation. Custom runtime parameters: Define parameters in a format that supports referencing platform-built-in variables or user-defined variables, enabling unified management of parameters for upstream and downstream tasks; Calculation result parameters: Parameters dynamically generated based on task execution results, obtained by writing specific calculation logic.

3. The method for dynamic parameter passing of tasks based on a big data execution engine according to claim 1, characterized in that, The types of upstream tasks include SparkSQL tasks, Python tasks, Shell tasks, and RDB tasks; SparkSQL task output parameters include: Before submitting tasks to the Spark cluster, the data platform performs format conversion on the output parameter statements and supplements the ORC storage format declaration and output path configuration. After receiving the task, the SparkSQL execution engine extracts the core query statement by matching the format using regular expressions, and writes the query results to the specified path in HDFS in ORC format; the platform will trigger automatic verification logic after the task is completed.

4. The method for dynamic parameter passing of tasks based on a big data execution engine according to claim 3, characterized in that, Python task output parameter implementation: Before submitting a Python task, the platform embeds the user-written output parameter statements into a pre-defined standardized script template. Dynamic configuration of environment parameters: The platform will add dedicated environment variables for each task to specify the name and storage path of the output file. The specific parameter configurations are as follows: The output file name should follow the same format as the SparkSQL task name; the parent directory of the output file; After the Python task is completed locally, the built-in file upload client is invoked to upload the locally generated parameter file to the corresponding HDFS directory based on the path information in the environment variables.

5. The method for dynamic parameter passing of tasks based on a big data execution engine according to claim 3, characterized in that, Shell task output parameter implementation: Before submitting the Shell task, the platform combines the user-configured output parameter statements with the pre-generated output path to form a complete writing script, which is then appended to the end of the user's original execution script; the output path format is consistent with that of the SparkSQL task. The Shell execution engine directly runs the concatenated script and writes the v1 string to the specified local path on the Linux server. If the local path does not exist, the platform will create it automatically; if an error occurs during the writing process, the task will terminate immediately and return an error code. If a downstream task needs to read this parameter file across servers, the platform will automatically synchronize the local file to shared storage or upload it directly to HDFS after the Shell task is executed.

6. The method for dynamic parameter passing of tasks based on a big data execution engine according to claim 3, characterized in that, RDB task output parameter implementation: Before submitting the RDB task, the SQL parsing engine is called to perform syntax validation and semantic analysis on the SQL statement corresponding to the output parameters; The platform encapsulates the parsed SQL statement into a standardized execution request structure and submits it to the RDB execution engine. The engine first executes the core business script written by the user. After the script executes successfully, it supplements the SQL statement corresponding to the execution parameters and obtains the query result set. Since the result set of the RDB task is a two-dimensional table structure, the platform converts it into a key-value pair format of row index-column name-value, and stores it in the platform's metadata database in the form of a JSON string, while also recording the upstream metadata corresponding to the parameters.

7. The method for dynamic parameter passing of tasks based on a big data execution engine according to claim 1, characterized in that, Based on the type of the upstream output parameters, a reference method adapted to the parameter type is used to retrieve the parameter values ​​from the corresponding storage medium and inject them into the execution context of the downstream task. Specifically, this includes: Obtaining constants and custom runtime parameters: Downstream tasks declare the upstream parameters they reference in the platform configuration interface, and the platform automatically records the relationships. Before the downstream task is executed, the platform calls the metadata database query interface to query the latest parsed values ​​of the upstream parameters based on the correlation; then the parameter values ​​are injected into the execution context of the downstream task in the form of environment variables or script variables. Before executing downstream tasks, the platform re-verifies the status of upstream parameters. If a parameter is missing or expired, the platform immediately terminates the execution of the downstream task and sends an alarm.

8. The method for dynamic parameter passing of tasks based on a big data execution engine according to claim 7, characterized in that, It also includes obtaining the parameters of the calculation results: If the upstream output is a calculation result parameter, the following reading strategy is adopted: Upstream parameter retrieval for SparkSQL / Python tasks: The computation results parameters of both SparkSQL and Python tasks are stored in HDFS as files. Downstream tasks obtain parameter values ​​by reading HDFS files and parsing them in a structured manner. Two reading modes are supported: row-column combined reading mode and columnar reading mode. Upstream parameter retrieval for Shell tasks: The computation results of Shell tasks are stored by default in the local path of the execution node. Downstream tasks select the corresponding reading method based on the file storage location, including: reading from the same node and reading from across nodes; upstream tasks retrieve parameters for RDB tasks. The computation results of RDB tasks are stored in the platform's metadata database in JSON format. Downstream tasks obtain these results through JSON parsing and structured references, including parameter querying and JSON parsing, structured references, and batch data processing.

9. The method for dynamic parameter passing of tasks based on a big data execution engine according to claim 1, characterized in that, Also includes: Through parameter version management, end-to-end monitoring, and three-level access control mechanisms, the traceability, reliability, and data security of parameter transmission are ensured. Specifically, this includes: Generate a unique version number for each output parameter; Real-time monitoring of the entire link status of parameters from upstream generation to downstream reference; control of parameter access based on a three-level permission system of tenant, project, and task.

10. A task dynamic parameter passing device based on a big data execution engine, wherein the task dynamic parameter passing method based on a big data execution engine according to any one of claims 1-9 is characterized in that, include: The parameter classification definition module is configured to be divided into three categories: explicit constant parameters, custom running parameters, and calculation result parameters, defining their data characteristics, configuration methods, and core uses respectively. The upstream task parameter output module is configured to handle four types of upstream tasks: SparkSQL, Python, Shell, and RDB. It uses processing logic adapted to the task type to store parameters in HDFS, local storage, or a metadata database. The downstream task parameter reference acquisition module is configured to first verify the upstream and downstream task dependency configurations and the availability status of upstream parameters, and then obtain the parameter values ​​according to the upstream parameter type and upstream task type and inject them into the downstream task execution context. The parameter transfer guarantee mechanism module is configured to ensure the traceability, reliability, and data security of parameter transfer through parameter version management, end-to-end status monitoring, and access control.

Images