An execution engine parameter recommendation method, device, equipment and storage medium

Abstract

The application discloses an execution engine parameter recommendation method and device, equipment and a storage medium, when a data processing task is received, metadata of a data object related to the data processing task is collected, and an operation type of the data processing task is parsed, an operation complexity coefficient is determined based on the operation type, and configuration parameters of a target execution engine are generated based on the metadata, the operation type and the operation complexity coefficient, so that the application utilizes specific metadata and semantic features related to the current data processing task to realize dynamic and fine-grained configuration parameter recommendation, the recommendation process is closely combined with physical and logical features of data, and the accuracy and universality of the recommendation result are improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and specifically provides a method, apparatus, device, and storage medium for recommending execution engine parameters. Background Technology

[0002] In the field of big data processing, the performance of data integration and computing tasks (such as data synchronization, ETL, ad-hoc queries, etc.) is highly dependent on the resource configuration parameters of the underlying computing engines (such as Apache Spark, Flink, DataX, etc.). Typical parameters include task parallelism (e.g., the number of Spark executors, the number of DataX channels), JVM heap memory size, and the number of CPU cores. These parameters directly affect task execution efficiency, resource utilization, and system stability. Currently, the configuration of these parameters generally relies on manual experience set by users. For example, when using DataX to synchronize data, users need to estimate the parallel channels and memory based on the size of the source table and network bandwidth; when submitting Spark SQL tasks, they need to estimate the required Executors, memory, and CPU resources based on the data volume and query complexity. Due to a lack of systematic understanding of data characteristics and task logic, users often find it difficult to accurately assess resource requirements, leading to problems such as insufficient resource allocation, excessive resource allocation, and high configuration barriers.

[0003] While some big data platforms offer default parameter templates or simple estimation suggestions based on historical tasks, their strategies are typically based on static rules or coarse-grained statistical information, making it difficult to achieve accurate and adaptive parameter recommendations. Existing technologies also involve two automated resource configuration schemes: tuning systems based on historical task performance feedback and static resource templates based on coarse-grained metadata. The core logic of tuning systems based on historical task performance feedback is to monitor execution metrics such as runtime, shuffle data volume, and GC time of historical tasks, and then use machine learning or heuristic rules to back-optimize resource configurations for subsequent similar tasks. However, this type of solution has significant shortcomings. It heavily relies on historical data, leading to a cold start dilemma when facing new tasks or significant changes in table structure. Furthermore, due to weak generalization ability, configuration deviations are prone to occur when there are significant differences in data scale and query logic between historical and new tasks. Additionally, optimization is lagging, requiring adjustments to be made only after the task is completed and metrics are collected, making it impossible to achieve proactive configuration before task submission. Static resource templates based on coarse-grained metadata, on the other hand, pre-set resource configuration templates according to different data sources or data volumes, directly matching the corresponding template based on the total data volume of the table when the task is created. The problems with this solution lie in its neglect of key information and rigidity of mechanisms. On the one hand, it only uses coarse-grained metadata such as the total amount of data at the table level, ignoring core factors that affect resource requirements, such as column cardinality, partition skew, storage format, and compression ratio. It also lacks awareness of SQL semantics and cannot identify high-overhead operations such as large table JOINs and complex aggregations, which can easily lead to insufficient resources for complex queries. On the other hand, the template threshold needs to be set manually, the rules are fixed and difficult to cover all business scenarios, and it does not have the ability to dynamically calculate and adaptively adjust.

[0004] In summary, existing execution engine parameter recommendation methods are disconnected from the physical and logical characteristics of the data, resulting in shortcomings in both accuracy and universality of the recommendation results. Summary of the Invention

[0005] This invention provides a method, apparatus, device, and storage medium for recommending execution engine parameters, which can improve the accuracy and universality of execution engine parameter recommendations.

[0006] In a first aspect, the present invention provides a method for recommending execution engine parameters, including: S1. In response to receiving a data processing task, collect the metadata of the data objects involved in the data processing task; S2. Analyze the operation type of the data processing task and determine the operation complexity coefficient based on the operation type; S3. Generate configuration parameters for the target execution engine based on the metadata, the operation type, and the operation complexity coefficient.

[0007] Further, step S3 includes: Based on the metadata, the operation type, and the operation complexity coefficient, the parallelism is calculated using a preset parallelism calculation formula, and the memory is calculated using a preset memory calculation formula. The configuration parameters include the parallelism and the memory.

[0008] Further, step S3 includes: The metadata, the operation type, and the operation complexity coefficient are input into a preset machine learning model, and the parallelism and memory are output. The configuration parameters include the parallelism and the memory.

[0009] Further, step S3 includes: Construct the objective function; Based on the metadata, the operation type, the operation complexity, and the objective function, the parallelism and memory are determined through heuristic search, and the configuration parameters include the parallelism and the memory.

[0010] Furthermore, after step S3, the execution engine parameter recommendation method further includes: The task configuration is generated based on the configuration parameters, and the task configuration is submitted to the target execution engine.

[0011] Furthermore, after submitting the task configuration to the target execution engine, the execution engine parameter recommendation method further includes: Real-time data collection of resource utilization rates when the target execution engine executes the data processing task; The actual resource requirement field in the preset metadata repository is updated based on the resource utilization rate. The actual resource requirement field is used to adjust the generated configuration parameters when the data processing task is received subsequently.

[0012] Furthermore, determining the operation complexity coefficient based on the operation type includes: Identify the high-overhead operation types from the given operation types; The operation complexity coefficient is determined based on the high-overhead operation type and the preset quantization rules.

[0013] Secondly, the present invention provides an execution engine parameter recommendation device, comprising: Metadata acquisition unit, used to acquire metadata of data objects involved in the data processing task in response to receiving a data processing task; A semantic parsing unit is used to parse the operation type of the data processing task and determine the operation complexity coefficient based on the operation type; The configuration parameter generation unit is used to generate configuration parameters for the target execution engine based on the metadata, the operation type, and the operation complexity coefficient.

[0014] Thirdly, the present invention also provides an execution engine parameter recommendation device, including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of any of the execution engine parameter recommendation methods provided in the first aspect of the present invention.

[0015] Fourthly, the present invention also provides a computer-readable storage medium comprising a computer program that, when run on an electronic device, causes the electronic device to perform the steps of any of the execution engine parameter recommendation methods provided in the first aspect of the present invention.

[0016] Beneficial effects The execution engine parameter recommendation method, apparatus, device, and storage medium provided by this invention collect metadata of the data objects involved in the data processing task when a data processing task is received, parse the operation type of the data processing task, determine the operation complexity coefficient based on the operation type, and then generate configuration parameters for the target execution engine based on the metadata, operation type, and operation complexity coefficient. Thus, this invention utilizes the specific metadata and semantic features involved in the current data processing task to achieve dynamic and fine-grained configuration parameter recommendation. The recommendation process is closely integrated with the physical and logical characteristics of the data, improving the accuracy and universality of the recommendation results. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating the execution engine parameter recommendation method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the execution engine parameter recommendation device provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of the execution engine parameter recommendation device provided in an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0020] It should be noted that if the embodiments of the present invention involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a specific posture. If the specific posture changes, the directional indicators will also change accordingly.

[0021] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Thus, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the use of "and / or" or "and / or" throughout the text implies three parallel solutions. For example, "A and / or B" includes solution A, solution B, or a solution where both A and B are satisfied. Furthermore, the technical solutions of the various embodiments can be combined, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0022] This invention discloses a method, apparatus, device, and storage medium for recommending execution engine parameters. To make the technical solution, purpose, and effects of this invention clearer, the specific implementation methods of this invention are described in detail below in conjunction with specific implementation scenarios. These implementation methods are intended to further explain the technical concept of this invention, rather than to limit the scope of protection of this invention. All equivalent modifications made based on the technical solution of this invention should be included within the scope of protection of this invention.

[0023] refer to Figure 1 This invention provides a method for recommending execution engine parameters, comprising the following steps: S1. In response to receiving a data processing task, collect the metadata of the data objects involved in the data processing task; S2. Analyze the operation type of the data processing task and determine the operation complexity coefficient based on the operation type; S3. Generate configuration parameters for the target execution engine based on metadata, operation type, and operation complexity coefficient.

[0024] For step S1, the data processing task is generally initiated by the user. For example, the user can create a data processing task on the platform interface, thereby initiating the data processing task.

[0025] For example, a user creates a data processing task: using the data acquisition tool DataX to synchronize data from a MySQL relational database to a data warehouse in Hive. This is done by submitting a Spark SQL query (such as SELECT a.id, COUNT( (FROM table_a a JOIN table_b b ON a.id = b.id GROUP BY a.id) initiates a data processing task.

[0026] The metadata of the data objects involved in the data processing task is table-level metadata, including the number of table rows, data size, column cardinality, partition structure, etc.

[0027] Specifically, embodiments of the present invention can query relevant metadata from a preset metadata repository based on table names. Optionally, the queried metadata may include the following physical characteristics: table_rows: Total number of rows in the table (unit: rows) data_size: Table storage size (in bytes) num_partitions: Number of partitions (if it is a partitioned table) column_cardinality: The cardinality of the key column (such as the JOIN column, GROUP BY column). storage_format: Storage format (Parquet / ORC / Text) compression_ratio: Compression ratio (used to estimate memory usage after decompression) More specifically, in this embodiment of the invention, static metadata and dynamic metadata are actively collected through a metadata proxy module. The static metadata is synchronized periodically from metadata repositories such as Hive Metastore and MySQL Information Schema, while the dynamic metadata is queried in real time when a data processing task is received.

[0028] The core of metadata-driven approaches lies in task context awareness—the recommendation parameters for the same table dynamically change under different queries (such as full table scan vs. partition filtering), while existing technologies only use static metadata at the table level. Compared to existing technologies, this invention combines the number of rows and the cardinality of columns after dynamic filtering to achieve precise task-level awareness.

[0029] Static metadata includes: table_rows: Total number of rows in the table (statistics obtained using ANALYZE TABLE) data_size: Table storage size (bytes, including partition summaries) storage_format: Storage format (Parquet / ORC / Text) partition_keys: List of partition fields Dynamic metadata includes: filtered_rows: Estimated number of rows after SQL predicate filtering (estimated using Hive Cost-Based Optimizer) join_keys_cardinality: The cardinality of the JOIN column (obtained via SHOW COLUMN STATS) To avoid interference from noisy data, some embodiments also employ preset filtering rules to filter the metadata. These filtering rules include: 1. Row count validity: If table_rows = 0 or the statistical period > n days (e.g., 7 days), then a real-time COUNT will be triggered. Sampling (e.g., sampling rate of 1%); 2. Column cardinality reliability: If the column cardinality is missing, it is estimated in real time using the HyperLogLog algorithm; 3. Partition pruning: Only collects metadata for partitions involved in the SQL WHERE conditions.

[0030] The mapping relationship between metadata and parameters is shown in Table 1: Table 1 For step S2, specifically, this embodiment of the invention performs semantic parsing on the data processing task to obtain the task logic of the data processing task, such as JOIN, GROUP BY, and DISTINCT in SQL, and quantifies them into operation complexity coefficients, such as JOIN=2.0 and GROUP BY=1.5. The task logic includes multiple operations, corresponding to multiple operation types.

[0031] It is understood that the embodiments of the present invention collect the metadata of the data objects involved in the data processing task and parse the operation type of the data processing task to determine the operation complexity coefficient. That is, by step S1 and step S2, the limitations of the existing execution engine parameter recommendation method that only relies on static data volume are overcome, so that parameter recommendation can be aware of computational complexity and avoid complex queries from failing due to insufficient configured memory.

[0032] Optionally, in some embodiments, determining the operation complexity coefficient based on the operation type in step S2 may include: Identify the high-overhead operation types from the operation types, and determine the operation complexity coefficient based on the high-overhead operation types and preset quantization rules.

[0033] In this embodiment, high-overhead operation types include JOIN, GROUP BY, and DISTINCT.

[0034] Methods for calculating operation complexity coefficients: in, , represents the value of the operation complexity coefficient when there are no high-overhead operation types; Indicates the first The weight of high-overhead operations (i.e., the operation complexity coefficient).

[0035] Specifically, the quantification rules (based on Spark execution plan analysis) are shown in Table 2: Table 2 In Table 2, the operation complexity coefficients were determined through actual testing using Spark UI (see Table 3), rather than being subjectively set.

[0036] Table 3 Optionally, in some embodiments, a dynamic weight adjustment mechanism is also included, which may include: If data skew is detected (e.g., the number of rows in a partition is more than 10 times the average), the operation complexity coefficient is multiplied by the first coefficient (e.g., 1.5). If Spark's Adaptive Query Execution (AQE) is enabled, the operation complexity coefficient is multiplied by a second coefficient (e.g., 0.8), achieving the effect of adaptively merging small tasks.

[0037] In this case, the second coefficient is less than the first coefficient; or, the first coefficient is greater than 1 and the second coefficient is less than 1.

[0038] Understandably, compared to existing technologies (such as IBM Adaptive Tuning) that rely solely on binary judgments of "whether JOIN is included", this embodiment quantifies the superposition effect of multiple operation combinations and achieves adaptive operation complexity coefficient adjustment based on operation type, making subsequent parameter configuration more accurate and reliable.

[0039] Furthermore, in this embodiment of the invention, after obtaining the metadata of the data objects involved in the data processing task, the operation type of the data processing task, and the corresponding operation complexity coefficient, a task feature vector can be extracted and constructed. The task feature vector F = [T, R, S, C, O]. Where T represents the task type (e.g., DataX, Spark, etc.), R represents the total number of rows, S represents the table storage data size (data_size), C represents the operation complexity coefficient, and O represents the optimization flag.

[0040] In task type T, DataX focuses on the number of channels and network bandwidth, while Spark focuses on Executor memory and Shuffle buffer. Therefore, a single engine in this embodiment can support multiple engines, reducing platform maintenance costs, and users do not need to learn the parameter tuning logic of different engines.

[0041] For step S3, based on the metadata, operation type, and operation complexity coefficient obtained in the previous steps, the recommended configuration parameters of the target execution engine can be calculated. There is no need to rely on human experience for configuration, which reduces resource waste and lowers the configuration threshold, significantly improving cluster resource utilization and task execution stability.

[0042] Optionally, in some embodiments, step S3 may include: Based on metadata, operation type, and operation complexity coefficient, the parallelism is calculated using a preset parallelism calculation formula, and the memory is calculated using a preset memory calculation formula. The configuration parameters include parallelism and memory.

[0043] Specifically, this embodiment can perform parallelism calculation based on the aforementioned task feature vector, combined with the parallelism calculation formula, and perform memory calculation based on the memory calculation formula.

[0044] The formula for calculating parallelism is: in, Indicates the degree of parallelism; This indicates the maximum parallelism of the cluster, which is a system configuration, such as the default of 200, and can be modified. This indicates the recommended number of rows to process per task, for example, the default is 1,000,000 rows, which can be modified.

[0045] Therefore, the physical meaning of the parallelism calculation formula can be understood: to ensure that each task processes a balanced amount of data and avoid small files or data skew.

[0046] Memory (JVM memory) calculation formula (based on JVM memory management model and storage format characteristics): in, This indicates the minimum memory requirement, for example, the default is 1GB, which can be modified. This represents the memory amplification factor, for example, Parquet=1.2, ORC=1.3, Text=2.0; This indicates the Shuffle buffer coefficient (including JOIN / GROUP BY). When the optimization flag AQE is enabled, ,otherwise .coefficient , It can be dynamically adjusted according to the task type T: when T=DataX, , When T=Spark, Determined based on storage format, Determined based on the operational complexity coefficient. Optionally, , The value of can be determined through offline stress testing experiments.

[0047] Therefore, we can understand the physical meaning of the memory calculation formula: the memory needs to cover the decompressed data and the Shuffle buffer.

[0048] Therefore, this embodiment calculates the parallelism based on metadata, operation type, and operation complexity coefficient using a parallelism calculation formula, and further calculates the memory usage using a memory calculation formula, thus obtaining the configuration parameters. Simultaneously, this embodiment achieves a precise mapping from data characteristics to physical resources (CPU core count, JVM memory), avoiding resource waste or memory overflow (OOM), and actual tests show an improvement in cluster throughput of over 30%.

[0049] Specifically, the derivation of the parallelism calculation formula is as follows: Parallelism directly impacts a task's concurrent execution capability and scheduling overhead. According to the official Apache Spark Tuning Guide (Apache Spark 3.3 Documentation), the recommended data size for a single task is between 100MB and 1GB to balance computational efficiency and scheduling overhead: tasks that are too small will significantly increase scheduling and serialization overhead, reducing throughput; tasks that are too large are prone to frequent garbage collection (GC) or out-of-memory (OOM) errors. Similarly, DataX community best practices recommend that a single channel process no more than 1 million rows to avoid memory pressure.

[0050] Based on the above principles, this embodiment of the invention sets the "ideal number of rows processed per task" as the configurable parameter Rper_task (default value is 1,000,000 rows). Considering that complex operations (such as JOIN, GROUP BY) may lead to intermediate data bloat, an operation complexity coefficient C is introduced to weight the original number of rows, resulting in the effective number of processed rows Reffective = R × C. Therefore, the theoretical parallelism is: To avoid excessive parallelism leading to cluster resource exhaustion or scheduling bottlenecks, a system parallelism limit is set. (Usually determined by the total number of CPU cores in the cluster). Finally, the parallelism is rounded and limited: This leads to the aforementioned formula for calculating parallelism. This formula achieves a dynamic balance between task granularity and cluster capability, ensuring high throughput and low overhead.

[0051] The derivation of the memory calculation formula is as follows: JVM memory requirements mainly consist of three parts: memory usage of decompressed data, Shuffle / Aggregation buffer overhead, and the JVM's own basic overhead. This invention establishes a memory computing model through the following steps: Step 1: Estimate the memory requirements for basic data Let the storage size of the table be S (in bytes, after compression), the operation complexity coefficient be C, and the parallelism be P. Then the average amount of data that each task needs to process is (S ⋅ C) / P.

[0052] Step 2: Consider the decompression amplification effect of the storage format. Columnar storage formats (such as Parquet and ORC) need to be decompressed into memory during reading, resulting in a significant space amplification effect. Based on actual tests of 100GB sample data (using Spark 3.2 + Snappy compression), the following typical amplification factors were obtained: Parquet's decompressed memory usage is approximately 1.2 times its compressed size; ORC (using Zlib compression) is approximately 1.3 times faster; Text formats (such as Gzip) can be magnified up to 2.0 times because they cannot be cropped by columns.

[0053] Therefore, a storage format magnification factor α is introduced to characterize this effect.

[0054] Step 3: Evaluate the memory overhead of the Shuffle operation When a task includes operations such as JOIN, GROUP BY, or DISTINCT, Spark needs to write intermediate data to a memory buffer during the Shuffle phase. According to Spark's Unified MemoryManager, the Shuffle memory overhead typically accounts for approximately 20% to 50% of the input data volume. Therefore, a Shuffle buffer factor β is introduced: When there is no shuffle operation, β = 0.2 (only for cache overhead). When there is a Shuffle operation, take β = 0.5 (including serialization and spill buffer).

[0055] Step 4: Establish a comprehensive memory model Taking all the above factors into account, the JVM heap memory required for a single Executor is: To prevent the JVM from failing to start due to insufficient memory configuration, a minimum memory threshold is set. (Default value is 1GB). The final memory configuration will be the larger of the two values: It should be noted that the specific values ​​of coefficients α and β can be calibrated through offline stress testing experiments, or dynamically adjusted according to the version of the execution engine, to ensure that the model can adapt to different operating environments.

[0056] Optionally, in some embodiments, step S3 may further include: Input metadata, operation type, and operation complexity coefficient into a preset machine learning model, and output parallelism and memory usage. Configuration parameters include parallelism and memory usage.

[0057] The machine learning model is pre-trained and can output the corresponding parallelism and memory configuration when given input metadata, operation type, and operation complexity coefficient, thus obtaining configuration parameters. For example, the machine learning model is a regression model, such as XGBoost or a neural network.

[0058] Optionally, the data input into the machine learning model may also include cluster status, such as CPU load and memory level.

[0059] It is understandable that this embodiment uses a machine learning model to replace rule formulas. While still based on metadata and task semantics, it has the following advantages: First, the model can capture the non-linear relationship between complex factors such as data skew and the impact of resources, and achieve more accurate demand prediction. Second, through an online learning mechanism, the model can continuously adapt to the environmental dynamics of a specific cluster and achieve continuous optimization of configuration strategies.

[0060] Optionally, in some embodiments, step S3 may further include: Construct the objective function; based on metadata, operation type, operation complexity, and the objective function, determine the parallelism and memory through heuristic search, and configure parameters including parallelism and memory.

[0061] Specifically, in this embodiment, the objective function is constructed, which can be to minimize the execution time or to minimize the resource cost. Then, based on metadata, operation type, operation complexity, and objective function, the optimal parallelism and memory combination are found through heuristic search, such as network search or Bayesian optimization, to obtain the configuration parameters.

[0062] For example, the objective function is: minP,M(ExecTime(P,M)+λ⋅ResourceCost(P,M)) Among them, ExecTime is estimated through historical task performance models, and λ is the cost weight.

[0063] It is understood that this embodiment achieves multi-objective dynamic trade-offs in resource allocation by using an optimized search strategy instead of traditional direct formula calculations. This embodiment can not only flexibly balance key indicators such as execution speed and resource cost, but also accurately adapt to service level agreement (SLA) sensitive scenarios, such as strictly limiting tasks to completion within a specified time window (e.g., within 30 minutes), thereby providing a reliable and efficient resource allocation solution under complex constraints.

[0064] Optionally, in some embodiments, to optimize the execution efficiency of tasks of different sizes, step S3 can also adopt a layered strategy to determine the configuration parameters. Specifically, this embodiment will perform layered processing based on the data volume: for small tasks less than 1GB, fixed low resources (such as parallelism P=1, memory M=1GB) are directly allocated to avoid the scheduling overhead caused by fine-grained tasks; for medium-sized tasks from 1GB to 1TB, the parallelism calculation formula and memory calculation formula are applied for precise calculation; and for large tasks exceeding 1TB, a data sharding awareness strategy is enabled to align its parallelism with the underlying storage (such as the number of HDFS blocks) to maximize I / O throughput. This layered logic is implemented by adding conditional judgments to the core process, and the scale-based layering further reduces scheduling overhead and improves the locality of large task data.

[0065] Furthermore, in some embodiments, after step S3, the method for recommending execution engine parameters may further include: The task configuration is generated based on the configuration parameters and then submitted to the target execution engine.

[0066] Specifically, in this embodiment, the configuration parameters determined in step S3 are written into the task configuration, for example: DataX: job.setting.speed.channel = P (task parallelism) Spark: spark.executor.instances = P (number of task executors) spark.executor.memory = M (task executor memory) Then, the task configuration is submitted to the target execution engine, so that the data processing task is submitted to the Spark / DataX cluster for execution with these configuration parameters, without the need for manual intervention by the user.

[0067] Furthermore, in some embodiments, after the step of "generating task configuration based on configuration parameters and submitting the task configuration to the target execution engine", the execution engine parameter recommendation method may further include: Real-time collection of resource utilization rate when the target execution engine performs data processing tasks, and updating the actual resource requirement field in the preset metadata repository based on the resource utilization rate.

[0068] The actual resource requirement field is used to adjust the generated configuration parameters when data processing tasks are received subsequently.

[0069] Therefore, when similar tasks arise later, resource configuration parameters can be dynamically adjusted and output based on the actual consumption records in the historical resource requirement database. Resource consumption metrics include one or more of the following: garbage collection (GC) time, shuffle overflow count, and actual memory usage.

[0070] This embodiment, while continuing the core concept of recommendation based on metadata, effectively overcomes the recommendation bias caused by the lag in updating metadata statistics by introducing a real-time feedback closed-loop mechanism. It also enables the system to adapt to changes in data distribution characteristics (such as sudden changes in the cardinality of a certain column of data), thereby upgrading the static forward prediction to a dynamic adaptive optimization system, significantly improving the accuracy of resource allocation and environmental adaptability.

[0071] Specifically, through research and practical testing, the embodiments of the present invention have the following advantages over existing execution engine parameter recommendation methods: 1) Resource utilization efficiency is significantly improved, directly saving computing costs, as shown in Table 3.

[0072] Table 3 2) Task execution efficiency and system throughput have been greatly improved, as shown in Table 4.

[0073] Table 4 3) The ease of operation has been revolutionized, and the technical threshold has been lowered, as shown in Table 5.

[0074] Table 5 4) Enhanced system stability and scalability, as shown in Table 6.

[0075] Table 6 5) Economic and environmental benefits, as shown in Table 7.

[0076] Table 7 In summary, the execution engine parameter recommendation method provided by this invention collects metadata of the data objects involved in the data processing task when a data processing task is received, parses the operation type of the data processing task, determines the operation complexity coefficient based on the operation type, and then generates configuration parameters for the target execution engine based on the metadata, operation type, and operation complexity coefficient. Thus, this invention utilizes the specific metadata and semantic features involved in the current data processing task to achieve dynamic and fine-grained configuration parameter recommendation. The recommendation process is closely integrated with the physical and logical characteristics of the data, improving the accuracy and universality of the recommendation results.

[0077] Specifically, this invention addresses the industry pain point of relying on manual experience, static templates, or delayed feedback for big data task resource configuration by constructing a data-aware, semantically understanding, and rule-driven intelligent parameter recommendation engine. This significantly lowers the user configuration threshold and precisely improves cluster resource utilization efficiency, enabling ordinary developers to use it efficiently without needing to master underlying engine tuning knowledge, avoiding resource waste and insufficiency, and ensuring stable and efficient task execution. Furthermore, this invention enhances the system's adaptability and generalization capabilities, effectively adapting to new tasks and data change scenarios. It also provides a unified parameter recommendation logic framework, scalable to support multiple computing engines such as DataX, Spark, and Flink, achieving standardized and intelligent configuration management.

[0078] Furthermore, this embodiment of the invention clearly divides the complete implementation process into two stages: "logical definition" and "physical implementation." In the first stage (step S1), medical personnel independently complete the business object modeling and indicator logic definition; in the second stage (steps S2-S4), developers are responsible for data source mapping and syntax translation. The clear boundaries between the two stages greatly reduce cross-domain communication costs and improve implementation efficiency.

[0079] refer to Figure 2 The present invention also provides an embodiment of an execution engine parameter recommendation apparatus for implementing the aforementioned execution engine parameter recommendation method, comprising: Metadata acquisition unit 201 is used to acquire metadata of data objects involved in the data processing task in response to receiving a data processing task; The semantic parsing unit 202 is used to parse the operation type of the data processing task and determine the operation complexity coefficient based on the operation type; The configuration parameter generation unit 203 is used to generate configuration parameters for the target execution engine based on the metadata, the operation type, and the operation complexity coefficient.

[0080] like Figure 3 As shown, Figure 3This is a schematic diagram of the structure of an execution engine parameter recommendation device provided in an embodiment of the present invention. The execution engine parameter recommendation device 1100 includes a processor 1101 with one or more processing cores, a memory 1102 with one or more computer-readable storage media, and a computer program stored in the memory 1102 and executable on the processor. The processor 1101 and the memory 1102 are electrically connected. Those skilled in the art will understand that the structure of the execution engine parameter recommendation device shown in the figure does not constitute a limitation on the execution engine parameter recommendation device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0081] The processor 1101 is the control center of the execution engine parameter recommendation device 1100. It connects to various parts of the execution engine parameter recommendation device 1100 via various interfaces and lines. By running or loading software programs and / or units stored in the memory 1102, and by calling data stored in the memory 1102, it executes various functions and processes data of the execution engine parameter recommendation device 1100, thereby providing overall monitoring of the execution engine parameter recommendation device 1100. The processor 1101 can be a CPU, GPU, network processor (NP), etc., and can implement or execute the methods, steps, and logic diagrams disclosed in the embodiments of this invention.

[0082] In this embodiment of the invention, the processor 1101 in the execution engine parameter recommendation device 1100 loads the instructions corresponding to the processes of one or more applications into the memory 1102 according to the following steps, and the processor 1101 runs the applications stored in the memory 1102 to achieve various functions. See the previous embodiments for details, which will not be repeated here.

[0083] Optional, such as Figure 3 As shown, the execution engine parameter recommendation device 1100 also includes: a touch display screen 1103, an radio frequency circuit 1104, an audio circuit 1105, an input unit 1106, and a power supply 1107. The processor 1101 is electrically connected to the touch display screen 1103, the radio frequency circuit 1104, the audio circuit 1105, the input unit 1106, and the power supply 1107. Those skilled in the art will understand that... Figure 3 The illustrated device structure for recommending execution engine parameters does not constitute a limitation on the device for recommending execution engine parameters. It may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0084] The touch display screen 1103 can be used to display a graphical user interface (GUI) and receive operation commands generated by the user interacting with the GUI. The touch display screen 1103 may include a display panel and a touch panel. The display panel can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces that recommend execution engine parameters. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. Optionally, the display panel can be configured using a liquid crystal display (LCD), organic light-emitting diode (OLED), or other similar technologies. The touch panel can be used to collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel), generate corresponding operation commands, and execute the corresponding program. Optionally, the touch panel may include a touch detection device and a touch controller. The touch detection device detects the user's touch location and the signal generated by the touch operation, transmitting the signal to the touch controller. The touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 1101. It can also receive and execute commands from the processor 1101. The touch panel can cover the display panel. When the touch panel detects a touch operation on or near it, it transmits the information to the processor 1101 to determine the type of touch event. Subsequently, the processor 1101 provides corresponding visual output on the display panel based on the type of touch event. In this embodiment, the touch panel and the display panel can be integrated into the touch display screen 1103 to achieve input and output functions. However, in some embodiments, the touch panel and the touch display screen 1103 can be used as two independent components to achieve input and output functions. That is, the touch display screen 1103 can also be used as part of the input unit 1106 to achieve input functions.

[0085] The radio frequency circuit 1104 can be used to transmit and receive radio frequency signals to establish wireless communication with network devices or other execution engine parameter recommendation devices via wireless communication, and to transmit and receive signals with network devices or other execution engine parameter recommendation devices.

[0086] Audio circuitry 1105 can be used to provide an audio interface between the user and the execution engine parameter recommendation device via a speaker and a microphone. Audio circuitry 1105 can convert received audio data into electrical signals and transmit them to the speaker, where the speaker converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by audio circuitry 1105, converted back into audio data, and then processed by processor 1101 before being transmitted via radio frequency circuitry 1104 to, for example, another execution engine parameter recommendation device, or output to memory 1102 for further processing. Audio circuitry 1105 may also include an earphone jack to provide communication between peripheral headphones and the execution engine parameter recommendation device.

[0087] The input unit 1106 can be used to receive input numbers, characters, or user characteristic information (such as fingerprints, iris, facial information, etc.), and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control.

[0088] Power supply 1107 is used to power the various components of execution engine parameter recommendation device 1100. Optionally, power supply 1107 can be logically connected to processor 1101 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. Power supply 1107 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0089] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0090] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0091] To this end, embodiments of the present invention provide a computer-readable storage medium storing a plurality of computer programs that can be loaded by a processor to execute any of the execution engine parameter recommendation methods provided in the embodiments of the present invention. The computer program can execute the steps of the aforementioned execution engine parameter recommendation method, as described in the preceding embodiments, and will not be repeated here.

[0092] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0093] Since the computer program stored in the computer-readable storage medium can execute any of the execution engine parameter recommendation methods provided in the embodiments of the present invention, the beneficial effects that any of the execution engine parameter recommendation methods provided in the embodiments of the present invention can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0094] In the above embodiments of the execution engine parameter recommendation device, computer-readable storage medium, execution engine parameter recommendation apparatus, and computer program product, the descriptions of each embodiment have different focuses. Parts not described in detail in a particular embodiment can be referred to in the relevant descriptions of other embodiments. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes and beneficial effects of the execution engine parameter recommendation device, computer-readable storage medium, computer program product, execution engine parameter recommendation apparatus, and their corresponding units described above can be referred to the description of the execution engine parameter recommendation method in the above embodiments, and will not be repeated here.

[0095] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural transformations made under the inventive concept of the present invention based on the description and drawings of the present invention, or direct / indirect applications of other related technical fields, are included within the patent protection scope of the present invention.

Claims

1. A method for recommending execution engine parameters, characterized in that, include: S1. In response to receiving a data processing task, collect the metadata of the data objects involved in the data processing task; S2. Analyze the operation type of the data processing task and determine the operation complexity coefficient based on the operation type; S3. Generate configuration parameters for the target execution engine based on the metadata, the operation type, and the operation complexity coefficient.

2. The execution engine parameter recommendation method according to claim 1, characterized in that, Step S3 includes: Based on the metadata, the operation type, and the operation complexity coefficient, the parallelism is calculated using a preset parallelism calculation formula, and the memory is calculated using a preset memory calculation formula. The configuration parameters include the parallelism and the memory.

3. The execution engine parameter recommendation method according to claim 1, characterized in that, Step S3 includes: The metadata, the operation type, and the operation complexity coefficient are input into a preset machine learning model, and the parallelism and memory are output. The configuration parameters include the parallelism and the memory.

4. The execution engine parameter recommendation method according to claim 1, characterized in that, Step S3 includes: Construct the objective function; Based on the metadata, the operation type, the operation complexity, and the objective function, the parallelism and memory are determined through heuristic search, and the configuration parameters include the parallelism and the memory.

5. The method for recommending execution engine parameters according to any one of claims 1-4, characterized in that, After step S3, the execution engine parameter recommendation method further includes: The task configuration is generated based on the configuration parameters, and the task configuration is submitted to the target execution engine.

6. The execution engine parameter recommendation method according to claim 5, characterized in that, After submitting the task configuration to the target execution engine, the execution engine parameter recommendation method further includes: Real-time data collection of resource utilization rates when the target execution engine executes the data processing task; The actual resource requirement field in the preset metadata repository is updated based on the resource utilization rate. The actual resource requirement field is used to adjust the generated configuration parameters when the data processing task is received subsequently.

7. The method for recommending execution engine parameters according to claim 1, characterized in that, The determination of the operation complexity coefficient based on the operation type includes: Identify the high-overhead operation types from the given operation types; The operation complexity coefficient is determined based on the high-overhead operation type and the preset quantization rules.

8. An execution engine parameter recommendation device, characterized in that, include: Metadata acquisition unit, used to acquire metadata of data objects involved in the data processing task in response to receiving a data processing task; A semantic parsing unit is used to parse the operation type of the data processing task and determine the operation complexity coefficient based on the operation type; The configuration parameter generation unit is used to generate configuration parameters for the target execution engine based on the metadata, the operation type, and the operation complexity coefficient.

9. An execution engine parameter recommendation device, characterized in that, It includes a processor and a memory, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the execution engine parameter recommendation method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a computer program that, when run on an electronic device, causes the electronic device to perform the steps of the execution engine parameter recommendation method according to any one of claims 1-7.

Images