Processing method and device for computing engine

Abstract

The invention discloses a processing method and device of a calculation engine, and belongs to the field of artificial intelligence. The method comprises the steps of obtaining a data calculation task, determining an initial task execution plan of the data calculation task through a pre-constructed intelligent decision model, verifying the data calculation task through a priority configuration rule in the intelligent decision model, and adjusting the initial task execution plan based on a verification result. The target task execution plan comprises the target engine identifier and the target resource configuration parameter, and based on the target resource configuration parameter, the target execution engine corresponding to the target engine identifier is adopted to process the data calculation task, so that the optimal task execution plan is determined through the intelligent decision-making model; the optimal execution engine and the optimal execution environment can be obtained through the task execution plan, the execution stability of the data calculation task is guaranteed, and the execution efficiency of the data calculation task is greatly improved.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence, and specifically relates to a processing method, apparatus, electronic device, and storage medium for a computing engine. Background Technology

[0002] In the digital age, with the rapid expansion of data scale, computing engines (such as the Spark engine) have become the core infrastructure for enterprises to realize data-driven business. However, early versions of computing engines suffered from problems such as low interpreted execution efficiency and JVM (Java Virtual Machine) performance bottlenecks. For example, Spark 2.0, despite introducing WholeStageCodegen technology, is still limited by JVM memory management, code generation mechanisms, and hardware instruction utilization.

[0003] Currently, to overcome JVM bottlenecks, the industry is turning to solutions based on vectorized execution and execution engines (such as native engines). However, existing architectures are rigid, tied to a single execution engine (such as using Velox or ClickHouse entirely within the Spark engine), making it impossible to select the optimal execution engine based on specific data computation tasks. Furthermore, execution engines like native engines introduce a large number of orthogonal parameters that require manual tuning, leading to an explosion of tuning combinations and making parameter configuration extremely prone to errors. Summary of the Invention

[0004] The purpose of this application is to provide a processing method and apparatus for a computing engine, which can solve the problems of rigid existing computing engine architectures, binding data computing tasks to a single execution engine, making it impossible to select the optimal execution engine according to the specific data computing task, and making it extremely easy to make mistakes in the configuration parameters for the execution engine.

[0005] To solve the above-mentioned technical problems, this application is implemented as follows: In a first aspect, embodiments of this application provide a processing method for a computing engine, characterized in that the computing engine includes multiple different execution engines, each execution engine having a corresponding engine identifier, the method comprising: Data acquisition and computation tasks; The initial task execution plan for the data computing task is determined by a pre-built intelligent decision-making model. The data computation task is verified using the priority configuration rules in the intelligent decision-making model, and the initial task execution plan is adjusted based on the verification results to obtain the target task execution plan; wherein, the target task execution plan includes the target engine identifier and the target resource configuration parameters; Based on the target resource configuration parameters, the target execution engine corresponding to the target engine identifier is used to process the data calculation task.

[0006] Optionally, the intelligent decision-making model includes an environmental perception module, and determining the initial task execution plan for the data computing task through the pre-built intelligent decision-making model includes: Based on the data calculation task, determine the first input feature for inputting into the intelligent decision-making model; The environmental perception module obtains second input features for inputting into the intelligent decision-making model. Based on the first input feature and the second input feature, an initial task execution plan for the data computation task is determined.

[0007] Optionally, the first input features include the target SQL operator and / or combination of target SQL operators corresponding to the data calculation task and the estimated data volume; The second input feature includes hardware information and resource usage information.

[0008] Optionally, the priority configuration rule includes multiple preset verification conditions, and the adjustment of the initial task execution plan based on the verification results to obtain the target task execution plan includes: If the data calculation task does not meet the multiple preset verification conditions, the initial task execution plan will be used as the target task execution plan. If the data calculation task meets at least one preset verification condition, the initial task execution plan is adjusted according to the priority configuration rule to obtain the target task execution plan.

[0009] Optionally, the priority configuration rule includes at least one first preset verification condition, at least one second preset verification condition, and at least one third preset verification condition. The first preset verification condition is used to verify the task characteristics of the data calculation task, the second preset verification condition is used to verify the target SQL operator of the data calculation task, and the third preset verification condition is used to verify the estimated data volume of the data calculation task. When the data calculation task meets the at least one preset verification condition, the initial task execution plan is adjusted according to the priority configuration rule to obtain the target task execution plan, including: If the data calculation task meets the first preset verification condition, the second engine identifier is determined based on the priority configuration rule; If the data calculation task meets the second preset verification condition, the second engine identifier is determined based on the priority configuration rule; If the data calculation task meets the third preset verification condition, the second resource configuration parameters are determined based on the priority configuration rules. The initial task execution plan is adjusted using the second engine identifier and / or the second resource configuration parameters to obtain the target task execution plan.

[0010] Optionally, the initial task execution plan includes a first engine identifier and a first resource configuration parameter. The step of adjusting the initial task execution plan using the second engine identifier and / or the second resource configuration parameter to obtain the target task execution plan includes: If the first engine identifier and the second engine identifier are inconsistent, the first engine identifier shall be replaced with the second engine identifier; If the first resource configuration parameter and the second resource configuration parameter are inconsistent, the first resource configuration parameter shall be replaced with the second resource configuration parameter; The adjusted initial task execution plan will be used as the target task execution plan.

[0011] Optionally, the intelligent decision-making model is constructed in the following manner: Obtain historical task information, as well as corresponding historical execution feedback information and historical resource usage information; Based on the historical task information and the historical execution feedback information, a computing engine analysis library is constructed; Based on the historical task information and the historical resource usage information, a configuration parameter analysis library is constructed. The intelligent decision-making model is obtained based on the computing engine analysis library and the configuration parameter analysis library.

[0012] Optionally, the step of constructing a computing engine analysis library based on the historical task information and the historical execution feedback information includes: Obtain each historical SQL operator and / or combination of historical SQL operators from the historical task information; Based on the historical execution feedback information, determine the recommendation engine identifier corresponding to each historical SQL operator and / or combination of historical SQL operators; The computational engine analysis library is constructed based on the various historical SQL operators and / or combinations of historical SQL operators and the corresponding recommendation engine identifiers.

[0013] Optionally, the step of constructing a configuration parameter analysis library based on the historical task information and the historical resource usage information includes: Obtain the combinations of historical SQL operators from the historical task information; Based on the historical resource usage information, the recommended configuration parameters corresponding to each historical SQL operator combination are determined; Based on the various historical SQL operator combinations and their corresponding recommended configuration parameters, a configuration parameter analysis library is constructed.

[0014] Secondly, embodiments of this application provide that the computing engine includes multiple different execution engines, each execution engine having a corresponding engine identifier, and the device includes: The task acquisition module is used to acquire data calculation tasks; The plan generation module is used to determine the initial task execution plan for the data computing task through a pre-built intelligent decision-making model. The plan optimization module is used to verify the data calculation task through the priority configuration rules in the intelligent decision model, and adjust the initial task execution plan based on the verification results to obtain the target task execution plan; wherein, the target task execution plan includes the target engine identifier and the target resource configuration parameters; The task processing module is used to process the data calculation task based on the target resource configuration parameters and the target execution engine corresponding to the target engine identifier.

[0015] In this embodiment, a data computing task is acquired, and an initial task execution plan is determined using a pre-built intelligent decision-making model. The data computing task is then validated using priority configuration rules within the intelligent decision-making model, and the initial task execution plan is adjusted based on the validation results to obtain a target task execution plan. The target task execution plan includes a target engine identifier and target resource configuration parameters. Based on the target resource configuration parameters, the target execution engine corresponding to the target engine identifier is used to process the data computing task. This achieves the determination of the optimal task execution plan based on a specific data computing task and through an experience-accumulated intelligent decision-making model. This avoids the limitation of using only a single execution engine and selecting a task execution plan based on static rules. The optimal execution engine and execution environment can be obtained through the task execution plan, ensuring the stability of the data computing task execution and significantly improving its execution efficiency. Attached Figure Description

[0016] To more clearly illustrate the technical solution of this application, the drawings used in the description of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1This is a flowchart of the processing steps of a computing engine provided in some embodiments of this application; Figure 2 This is a schematic diagram of the execution flow of the Sparknative intelligent engine provided in some embodiments of this application; Figure 3 This is a structural block diagram of an execution device for a computing engine provided in some embodiments of this application; Figure 4 This is a schematic diagram of the hardware structure of an electronic device provided in some embodiments of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0020] In today's digital age, the scale and complexity of data are exploding. Enterprises and organizations need to process massive amounts of data to extract valuable information to support decision-making, business optimization, and innovation. For example, the Spark engine, with its unified computing model, hardware acceleration optimization, and ecosystem integration capabilities, has become a core engine for enterprises to build data-driven systems.

[0021] Early versions of the Spark engine used an interpreted execution model, which suffered from frequent function calls, poor CPU (Central Processing Unit) caching, and difficulties in compiler optimization. Spark 2.0 introduced WholeStageCodegen technology, which reduced function call overhead, but it was still limited by JVM performance.

[0022] JVM performance bottlenecks are mainly manifested in the following aspects: 1. Memory management performance bottleneck: The JVM heap memory management mode causes the GC (Garbage Collection) pause time to increase non-linearly with the data scale, and cannot avoid the catastrophic impact of Full GC on large-scale Shuffle tasks.

[0023] 2. Defects in code generation mechanism: Although WholeStageCodegen optimizes intermediate code through LLVM (Low-Level Virtual Machine), it is limited by the 64KB bytecode limit of JVM methods. Complex operators are prone to triggering a fallback to interpreted execution mode, resulting in performance degradation.

[0024] 3. Low hardware instruction utilization: Modern CPUs' SIMD (Single Instruction Multiple Data) vectorized instruction set can only be indirectly called in the JVM through JNI (Java Native Interface), and the instruction pipeline utilization is less than 40% of that of the native C++ implementation.

[0025] To address JVM performance bottlenecks and fully utilize CPU performance, vectorized execution and native engines have become a key area for breakthrough. Several solutions are now available in the industry. However, with the introduction and large-scale production use of the Spark Native engine, other issues have arisen, such as: 1. Dilemmas and architectural contradictions in selecting native engine technologies Currently, mainstream native engines exhibit significant differences in their technical implementation. For example, Velox and ClickHouse have conflicting technical choices. Existing architectures only support binding to a single backend engine and cannot achieve dynamic routing based on a cost model. For instance, routing Scan+Filter to ClickHouse and multi-table Join to Velox leads to increased resource utilization losses.

[0026] 2. Defects of the Fallback Mechanism The existing solution adopts an all-or-nothing strategy, which cannot identify the scenarios supported by some operators in the hybrid execution plan (such as 80% of SQL operators can be executed natively), resulting in a potential performance loss of 40%.

[0027] 3. The exponential increase in the complexity of parameter tuning The native engine introduces a multi-dimensional orthogonal parameter system, resulting in an explosion of optimization combinations: Memory coordination management challenges: It is necessary to balance the ratio of JVMHeap (heap memory), Off-Heap (off-heap memory) and Native memory pool. Incorrect configuration can easily lead to OOM (OutOfMemory) or memory fragmentation.

[0028] Hardware instruction set adaptation parameters: SIMD bit width thresholds need to be set for different CPU architectures (such as x86AVX2 / AVX-512, ARMSVE2). Incorrect configuration will lead to a decrease in instruction throughput.

[0029] Shuffle optimization parameters: Fragmented columnar Shuffle requires independent configuration of multiple parameters, including compression algorithm (such as ZSTD, Snappy) and batch size.

[0030] The following description, in conjunction with the accompanying drawings, details a processing method for a computing engine provided in this application through specific embodiments and application scenarios.

[0031] Reference Figure 1 This document illustrates a flowchart of the processing steps of a computing engine according to some embodiments of this application. The computing engine includes multiple different execution engines, each with a corresponding engine identifier. Specifically, the method may include the following steps: Step 101: Obtain data calculation tasks.

[0032] In the embodiments of this application, the SparkNative engine is used as the computing engine to illustrate the specific implementation of this solution. The SparkNative engine may include multiple execution engines, each of which has a corresponding engine identifier. The multiple execution engines may include JVM and Native engines (such as Velox and ClickHouse).

[0033] Specifically, the Spark Native engine obtains the data computation task to be processed, which includes the target SQL statement. In practical applications, the SQL statement can be converted into multiple SQL operators and / or combinations of SQL operators.

[0034] Step 102: Determine the initial task execution plan for the data computing task using a pre-built intelligent decision-making model.

[0035] Specifically, to determine how to process the acquired data computation tasks, a pre-built intelligent decision-making model can be used to determine the initial task execution plan. This intelligent decision-making model is built on machine learning and uses Spring to construct microservices, providing a service port for calling the model. The Spark driver can call this microservice port to obtain the task execution plan for the data computation tasks.

[0036] In some embodiments of this application, the intelligent decision-making model is constructed in the following manner: Step 11: Obtain historical task information, as well as corresponding historical execution feedback information and historical resource usage information.

[0037] To build an intelligent decision-making model, we can first obtain historical task information, which includes multiple historical data calculation tasks. We also need to obtain the corresponding historical execution feedback information and historical resource usage information, that is, the feedback information after the execution of multiple historical data calculation tasks, and the historical resource usage information corresponding to the execution of historical calculation tasks.

[0038] Historical execution feedback information refers to the execution effect feedback information of each SQL operator and / or combination of SQL operators corresponding to the SQL statements in multiple historical data calculation tasks. This allows us to identify the execution engine that performs best for each SQL operator or combination of SQL operators. Historical resource usage information refers to JVMHeap, Off-Heap, and Native memory allocation information, SIMD bit width threshold, compression algorithm, batch size, etc., when executing multiple historical data calculation tasks.

[0039] Step 12: Construct a computing engine analysis library based on the historical task information and the historical execution feedback information.

[0040] Specifically, a computing engine analysis library can be built based on multiple historical data computing tasks in the historical task information and the corresponding historical execution feedback information. This computing engine analysis library is used to recommend the optimal execution engine for data computing tasks.

[0041] Step 13: Construct a configuration parameter analysis library based on the historical task information and the historical resource usage information.

[0042] Specifically, the historical resource usage information is first normalized, and then a configuration parameter analysis library is constructed based on multiple historical data calculation tasks in the historical task information and the corresponding historical resource usage information. This configuration parameter analysis library is used to recommend resource configuration parameters for data calculation tasks.

[0043] Step 14: Based on the computing engine analysis library and configuration parameter analysis library, the intelligent decision-making model is obtained.

[0044] After obtaining the computing engine analysis library and the configuration parameter analysis library, intelligent decision-making models can be built based on these two analysis libraries.

[0045] In some embodiments of this application, the step of constructing a computing engine analysis library based on the historical task information and the historical execution feedback information includes: Sub-step 21: Obtain each historical SQL operator and / or combination of historical SQL operators from the historical task information.

[0046] Specifically, the historical SQL statements of each historical data calculation task in the historical task information can be parsed into historical SQL operators and / or combinations of historical SQL operators.

[0047] Sub-step 21: Based on the historical execution feedback information, determine the recommendation engine identifier corresponding to each historical SQL operator and / or combination of historical SQL operators.

[0048] After parsing the SQL statements of each historical data calculation task in the historical task information into SQL operators and / or combinations of SQL operators, a performance baseline can be established for the execution of each SQL operator and / or combination of SQL operators, mainly including single table scans, joins, and aggregation calculations. Then, based on the acquired historical task information and historical execution feedback information, recommended engine identifiers can be marked for each SQL operator and / or combination of SQL operators according to the established performance baselines.

[0049] Sub-step 23: Based on the various historical SQL operators and / or combinations of historical SQL operators and the corresponding recommendation engine identifiers, construct the computing engine analysis library.

[0050] After labeling the recommended engine identifiers for each SQL operator and / or combination of SQL operators, the computing engine analysis library can be constructed based on the historical SQL operators and / or combinations of historical SQL operators and their corresponding recommended engine identifiers.

[0051] In some embodiments of this application, the step of constructing a configuration parameter analysis library based on the historical task information and the historical resource usage information includes: Sub-step 31: Obtain the combinations of historical SQL operators from the historical task information.

[0052] Specifically, the historical SQL statements of each historical data calculation task in the historical task information can be parsed into combinations of historical SQL operators.

[0053] Sub-step 32: Based on the historical resource usage information, determine the recommended configuration parameters corresponding to each historical SQL operator combination.

[0054] After obtaining the historical SQL operator combinations for each historical data calculation task, the optimal resource configuration parameters for each historical SQL operator combination can be marked by combining the corresponding historical resource usage information. These recommended configuration parameters can be a combination of parameters.

[0055] Sub-step 33: Based on the various historical SQL operator combinations and the corresponding recommended configuration parameters, construct a configuration parameter analysis library.

[0056] Finally, a configuration parameter analysis library is built based on various historical SQL operator combinations and corresponding recommended configuration parameters.

[0057] In addition, when building an intelligent decision-making model based on the computing engine analysis library and configuration parameter analysis library, a priority configuration rule library and an environment perception module can also be constructed.

[0058] The priority rule configuration library includes priority configuration rules, which can include native engine priority selection rules, fallback priority rules, and configuration parameter priority selection rules. That is, you can construct native engine priority selection rules, fallback priority rules, and configuration parameter priority selection rules to obtain priority configuration rules, and then build the priority configuration rule library based on these priority configuration rules. The native engine priority selection rules, fallback priority rules, and configuration parameter priority rules will be explained in detail later.

[0059] The environment awareness module is mainly used for hardware topology awareness and data feature analysis of the engine cluster. Hardware topology awareness can include the overall CPU rate of the cluster and each node, and the memory usage of the cluster and each node. Data feature analysis can include real-time data feedback such as shuffle data volume and data compression ratio.

[0060] Furthermore, an intelligent decision-making model is constructed using a computational engine analysis library, a configuration parameter analysis library, a priority configuration rule library, and an environmental perception module. After obtaining the intelligent decision-making model, it can be trained; specific training methods are not elaborated here, as model training is a standard technique in this field.

[0061] After training, the intelligent decision-making model can be evaluated and validated, such as through basic validation using TPC-DS standardized validation, or through non-standardized validation using extreme cases. Specifically: 1. Standardized Validation Example of TPC-DS (Transaction Processing Performance Council-Data Support, Multi-table Join Aggregation Query) Based on TPC-DS, a three-table Join (join), GROUP BY (grouping), and SUM / AVG (summation / average) aggregation query was used to verify the engine's dynamic selection (multi-table Join routing Velox) and parameter adaptation effects.

[0062] For a large-scale scan of the catalog_sales table (TPC-DS100GB), combined with multi-condition filtering and sorting queries, the performance of vectorized execution and hardware instruction set compatibility were verified.

[0063] 2. Example of non-standardized verification in extreme cases A GroupBy query was used to verify the Fallback strategy and optimization of skew handling parameters in a data skew scenario, where 90% of the data in a single table is concentrated on a single key.

[0064] A 5TB-level cross-node Shuffle multi-table join query was performed, with cluster memory utilization reaching 85%, validating the stability of memory collaborative management and dynamic adjustment of Shuffle parameters.

[0065] In some embodiments of this application, the intelligent decision-making model includes an environmental perception module, and determining the initial task execution plan for the data computing task through a pre-built intelligent decision-making model includes: Sub-step 41: Based on the data calculation task, determine the first input feature to be used as input to the intelligent decision-making model.

[0066] The input to the intelligent decision-making model can be SQL operators, combinations of SQL operators, estimated data volume, hardware information, and current resource usage information. Hardware information can include node CPU / memory / disk configuration, the number of cluster nodes, and the topology. Current resource usage information can include dynamic resources, task context, and data distribution information. Dynamic resources can include the real-time CPU / memory / network / disk load of the engine cluster and nodes. Task context can include the number of currently running tasks, queue priorities, and resource quotas. Data distribution information can include data locality, Shuffle data volume, and storage format.

[0067] Therefore, based on the data computation task, the first input feature used to input the intelligent decision-making model can be determined. This first input feature may include the target SQL operator and / or combination of target SQL operators corresponding to the data computation task, as well as the estimated data volume.

[0068] Sub-step 42: The environmental perception module obtains the second input features used to input the intelligent decision-making model.

[0069] While acquiring the first input features, the environmental perception module can also acquire a second input feature for inputting into the intelligent decision-making model. This second input feature may include hardware information and current resource usage information.

[0070] Sub-step 42: Determine the initial task execution plan for the data computation task based on the first input feature and the second input feature.

[0071] The target SQL operator and / or combination of target SQL operators, estimated data volume, hardware information, and current resource usage information are then input into the intelligent decision-making model to determine the initial task execution plan for the data computing task.

[0072] Step 103: Verify the data computation task using the priority configuration rules in the intelligent decision-making model, and adjust the initial task execution plan based on the verification results to obtain the target task execution plan; wherein, the target task execution plan includes the target engine identifier and target resource configuration parameters.

[0073] The initial task execution plan may include a first engine identifier and first resource configuration parameters, while the target task execution plan includes a target engine identifier and target resource configuration parameters. After obtaining the initial task execution plan, the data computation task can be validated using the priority configuration rules in the aforementioned priority configuration rule base. Based on the validation results, the first engine identifier and first resource configuration parameters in the initial task execution plan are adjusted to obtain the target task execution plan.

[0074] In some embodiments of this application, the priority configuration rule includes multiple preset verification conditions, and the adjustment of the initial task execution plan based on the verification results to obtain the target task execution plan includes: Sub-step 51: If the data calculation task does not meet the multiple preset verification conditions, the initial task execution plan is used as the target task execution plan.

[0075] The priority configuration rules include Native engine priority selection rules, Fallback priority rules, and configuration parameter priority selection rules, which correspond to multiple preset verification conditions. If the data calculation task does not meet any of the preset verification conditions, then the initial task execution plan is directly used as the target task execution plan, that is, the first engine identifier and the first resource configuration parameter are used as the target engine identifier and the target resource configuration parameter.

[0076] Sub-step 52: If the data calculation task meets at least one preset verification condition, the initial task execution plan is adjusted according to the priority configuration rule to obtain the target task execution plan.

[0077] If the data computation task meets at least one preset verification condition, that is, the data task may need to prioritize the corresponding configuration for special cases, that is, it is necessary to adjust the first engine identifier and the first resource configuration parameter in the initial task execution plan through the priority configuration rule to obtain the target task execution plan.

[0078] The intelligent decision-making model can be understood as consisting of two parts: an intelligent dynamic model (computation engine analysis library and configuration parameter analysis library) and a priority configuration rule model (priority configuration rule library). The priority configuration rule model will rely on past experience to form specific executable rules. If there is a conflict between the priority configuration rule model and the intelligent dynamic model, the priority configuration rule model will be used to determine the priority.

[0079] In some embodiments of this application, the priority configuration rule includes at least one first preset verification condition, at least one second preset verification condition, and at least one third preset verification condition. The first preset verification condition is used to verify the task characteristics of the data computation task, the second preset verification condition is used to verify the target SQL operator of the data computation task, and the third preset verification condition is used to verify the estimated data volume of the data computation task. When the data computation task satisfies the at least one preset verification condition, the initial task execution plan is adjusted according to the priority configuration rule to obtain the target task execution plan, including: Sub-step 61: If the data calculation task meets the first preset verification condition, determine the second engine identifier based on the priority configuration rule.

[0080] The first preset verification condition is the verification condition corresponding to the Native engine priority selection rule, which is used to verify the task characteristics of the data computing task. If the data computing task meets the first preset verification condition, the second engine identifier is determined by the Native engine priority selection rule in the priority configuration rule.

[0081] As an example, when the SQL operator combination of the SQL statement to be executed is an INNERJOIN + complex aggregation (including GROUP BY + SUM / AVG multi-dimensional calculations) of 3 or more large tables (single table data volume ≥ 20GB), and the CPU of the engine cluster node supports the AVX-512 instruction set and the memory utilization is ≤ 60%, the Velox engine is preferred as the execution engine. The reason is that Velox has more mature optimization for vectorized HashJoin of multi-table joins, can efficiently handle intermediate data after large table joins, and its memory management mechanism can adapt to the memory fluctuation requirements of complex aggregations. At the same time, the adaptation of the AVX-512 instruction set can further improve the computing throughput, while ClickHouse is prone to Shuffle bottlenecks in multi-table complex join scenarios.

[0082] Sub-step 62: If the data calculation task meets the second preset verification condition, determine the second engine identifier based on the priority configuration rule.

[0083] The second preset verification condition is the verification condition corresponding to the Fallback priority rule, which is used to verify the target SQL operator of the data calculation task. If the data calculation task meets the second preset verification condition, the second engine identifier is determined by the Fallback priority rule in the priority configuration rule.

[0084] As an example, when data processing tasks reveal that 85% of SQL operators (such as Scan, Filter, and simple Aggregator) support native engine execution, while only 15% of operators (such as special window functions and user-defined functions) do not support native execution, a "partial fallback" strategy is adopted. This means that operators supporting native execution retain the native execution mode, while unsupported operators fall back to JVM execution. Furthermore, intermediate results from native execution are reused through data sharding to avoid performance loss caused by a full fallback. If the proportion of unsupported operators is ≥50%, or if the data transfer cost after fallback exceeds the benefits of native execution (e.g., intermediate result data volume ≥100GB), then the system switches to full JVM execution mode.

[0085] Sub-step 63: If the data calculation task meets the third preset verification condition, determine the second resource configuration parameters based on the priority configuration rule.

[0086] The third preset verification condition is the verification condition corresponding to the configuration parameter priority selection rule, which is used to verify the estimated data volume of the data calculation task. When the data calculation task meets the third preset verification condition, the second resource configuration parameter is determined by the configuration parameter priority selection rule in the priority configuration rule.

[0087] As an example, when the SQL statement contains large-scale Shuffle operations (Shuffle data volume ≥ 50GB), and the cluster's current memory utilization is ≥ 70% and CPU utilization is ≤ 50%, the parameter selection rules are as follows: Set the Native memory pool ratio to 30% of physical memory (lower than the default 50%) to avoid competing for resources with JVM heap memory and off-heap memory; select ZSTD as the Shuffle compression algorithm (instead of the default Snappy) to reduce the amount of Shuffle data transfer at the cost of a small amount of CPU computing power; adjust the batch size to 8MB (default 4MB) to reduce the number of IO operations during the Shuffle stage. If the cluster memory utilization is ≤ 50% and CPU utilization is ≥ 80%, switch to the Snappy compression algorithm + 4MB batch size to prioritize CPU processing efficiency.

[0088] Sub-step 64: Adjust the initial task execution plan using the second engine identifier and / or the second resource configuration parameters to obtain the target task execution plan.

[0089] After obtaining the second engine identifier and / or second resource configuration parameters using the priority configuration rules, the first engine identifier and / or first resource configuration parameters in the initial task execution plan are adjusted using the second engine identifier and / or second resource configuration parameters to obtain the target task execution plan.

[0090] In some embodiments of this application, the initial task execution plan includes a first engine identifier and a first resource configuration parameter. The step of adjusting the initial task execution plan using the second engine identifier and / or the second resource configuration parameter to obtain a target task execution plan includes: Sub-step 71: If the first engine identifier and the second engine identifier are inconsistent, replace the first engine identifier with the second engine identifier.

[0091] If the first engine identifier and the second engine identifier are inconsistent, the first engine identifier will be replaced with the second engine identifier. That is, if there is a conflict between the priority configuration rule model and the intelligent dynamic model, the priority configuration rule model will take precedence. If they are consistent, no replacement is needed.

[0092] Sub-step 72: If the first resource configuration parameter and the second resource configuration parameter are inconsistent, replace the first resource configuration parameter with the second resource configuration parameter.

[0093] If the first resource configuration parameter and the second resource configuration parameter are inconsistent, replace the first resource configuration parameter with the second resource configuration parameter. If they are consistent, no replacement is needed.

[0094] Sub-step 73: Use the adjusted initial task execution plan as the target task execution plan.

[0095] After the replacement is completed, the adjusted initial task execution plan can be used as the target task execution plan.

[0096] Step 104: Based on the target resource configuration parameters, the data computation task is processed using the target execution engine corresponding to the target engine identifier.

[0097] After obtaining the target task execution plan, you can configure it based on the target resource configuration parameters in the target task execution plan, and then use the target execution engine corresponding to the target engine identifier in the target task execution plan to process the data calculation task.

[0098] In practical applications, specific task processing information can be integrated and written into a database as training and testing data for subsequent intelligent decision-making models. Furthermore, the intelligent decision-making model takes multiple dimensions of data as input, including SQL operators, resource information, shuffle metrics, and other SQL execution metrics and parameters. By continuously iterating the model through data calculation of task processing information, the selection of task execution plans can be made more accurate.

[0099] In this embodiment, a data computing task is acquired, and an initial task execution plan is determined using a pre-built intelligent decision-making model. The data computing task is then validated using priority configuration rules within the intelligent decision-making model, and the initial task execution plan is adjusted based on the validation results to obtain a target task execution plan. The target task execution plan includes a target engine identifier and target resource configuration parameters. Based on the target resource configuration parameters, the target execution engine corresponding to the target engine identifier is used to process the data computing task. This achieves the determination of the optimal task execution plan based on a specific data computing task and through an experience-accumulated intelligent decision-making model. This avoids the limitation of using only a single execution engine and selecting a task execution plan based on static rules. The optimal execution engine and execution environment can be obtained through the task execution plan, ensuring the stability of the data computing task execution and significantly improving its execution efficiency.

[0100] Reference Figure 2 This document illustrates a flowchart of the execution process of a Spark Native intelligent engine according to some embodiments of this application, which may specifically include the following steps: 1. The SQL statements for data computation tasks are input into the Spark driver of the Spark engine, yielding SQL operators and / or combinations of SQL operators, along with an estimated data volume. The environment awareness module of the intelligent decision-making model acquires hardware information and current resource usage information (hardware and software information), and inputs this information into the intelligent decision-making model. Specifically, the intelligent decision-making model constructs and trains itself using historical execution feedback information from historical task processing information in the knowledge base, historical data computation tasks, and historical resource usage information.

[0101] 2. The intelligent decision model determines the target task execution plan through the intelligent decision model service (i.e., the intelligent dynamic model) and the rule model (i.e., the priority configuration rule model) and returns it to the Spark driver.

[0102] 3. According to the target task execution plan, process the SQL statements for data calculation tasks. First, input them into the glue layer for translation or adaptation, then input them into the native execution layer. Based on the target task execution plan, determine the target execution engine for processing. After processing, execute metrics to record task processing information into the database for training and iteration of the intelligent decision-making model.

[0103] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of this application.

[0104] Reference Figure 3 The diagram illustrates the structure of an execution device for a computing engine according to some embodiments of this application. The computing engine includes multiple different execution engines, each with a corresponding engine identifier, and may specifically include the following modules: Task acquisition module 301 is used to acquire data calculation tasks; The plan generation module 302 is used to determine the initial task execution plan of the data computing task through a pre-built intelligent decision model; The plan optimization module 303 is used to verify the data calculation task through the priority configuration rules in the intelligent decision model, and adjust the initial task execution plan based on the verification results to obtain the target task execution plan; wherein, the target task execution plan includes the target engine identifier and the target resource configuration parameters; The task processing module 304 is used to process the data calculation task based on the target resource configuration parameters and the target execution engine corresponding to the target engine identifier.

[0105] In one embodiment of this application, the intelligent decision-making model includes an environmental perception module, and the plan generation module 302 includes: The first feature determination submodule is used to determine the first input feature for inputting the intelligent decision model based on the data calculation task. The second feature acquisition submodule is used to acquire second input features for inputting into the intelligent decision-making model through the environment perception module; The task determination submodule is used to determine the initial task execution plan for the data calculation task based on the first input feature and the second input feature.

[0106] In one embodiment of this application, the first input feature includes the target SQL operator and / or combination of target SQL operators corresponding to the data calculation task and the estimated data volume; The second input feature includes hardware information and resource usage information.

[0107] In one embodiment of this application, the priority configuration rule includes multiple preset verification conditions, and the plan optimization module 303 includes: The target task determination submodule is used to take the initial task execution plan as the target task execution plan when the data calculation task does not meet the multiple preset verification conditions. The adjustment submodule is used to adjust the initial task execution plan according to the priority configuration rules to obtain the target task execution plan when the data calculation task meets at least one preset verification condition.

[0108] In one embodiment of this application, the priority configuration rule includes at least one first preset verification condition, at least one second preset verification condition, and at least one third preset verification condition. The first preset verification condition is used to verify the task characteristics of the data calculation task, the second preset verification condition is used to verify the target SQL operator of the data calculation task, and the third preset verification condition is used to verify the estimated data volume of the data calculation task. The adjustment submodule includes: The first verification unit is used to determine the second engine identifier based on the priority configuration rule when the data calculation task meets the first preset verification condition. The second verification unit is used to determine the second engine identifier based on the priority configuration rule when the data calculation task meets the second preset verification condition. The third verification unit is used to determine the second resource configuration parameters based on the priority configuration rules when the data calculation task meets the third preset verification conditions. The target task determination unit is used to adjust the initial task execution plan through the second engine identifier and / or the second resource configuration parameters to obtain the target task execution plan.

[0109] In one embodiment of this application, the initial task execution plan includes a first engine identifier and a first resource configuration parameter, and the target task determination unit includes: The first replacement subunit is used to replace the first engine identifier with the second engine identifier when the first engine identifier and the second engine identifier are inconsistent. The second replacement subunit is used to replace the first resource configuration parameter with the second resource configuration parameter when the first resource configuration parameter and the second resource configuration parameter are inconsistent. The target task determination sub-unit is used to take the adjusted initial task execution plan as the target task execution plan.

[0110] In one embodiment of this application, the intelligent decision-making model is constructed using the following modules: The historical information acquisition module is used to acquire historical task information, as well as corresponding historical execution feedback information and historical resource usage information; The first analysis library construction module is used to construct a computing engine analysis library based on the historical task information and the historical execution feedback information. The second analysis library construction module is used to construct a configuration parameter analysis library based on the historical task information and the historical resource usage information. The model building module is used to obtain the intelligent decision-making model based on the computing engine analysis library and the configuration parameter analysis library.

[0111] In one embodiment of this application, the first analysis library construction module includes: The first information parsing submodule is used to obtain each historical SQL operator and / or combination of historical SQL operators in the historical task information; The feedback information analysis submodule is used to determine the recommendation engine identifier corresponding to each historical SQL operator and / or combination of historical SQL operators based on the historical execution feedback information. The first analysis library construction submodule is used to construct the computing engine analysis library based on the various historical SQL operators and / or combinations of historical SQL operators and the corresponding recommendation engine identifiers.

[0112] In one embodiment of this application, the root second analysis library construction module includes: The second information parsing submodule is used to obtain the combinations of historical SQL operators in the historical task information; The information analysis submodule is used to determine the recommended configuration parameters corresponding to each historical SQL operator combination based on the historical resource usage information. The second analysis library construction submodule is used to construct a configuration parameter analysis library based on the various historical SQL operator combinations and the corresponding recommended configuration parameters.

[0113] The execution device of a computing engine in this application embodiment can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not impose specific limitations.

[0114] The execution device of a computing engine in this application embodiment can be a device with an operating system. The operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0115] The execution device of a computing engine provided in this application embodiment can achieve... Figures 1 to 2 The various processes implemented by the execution device of a computing engine in the method embodiment are not described in detail here to avoid repetition.

[0116] Optionally, this application embodiment also provides an electronic device, including a processor 410, a memory 409, and a program or instructions stored in the memory 409 and executable on the processor 410. When the program or instructions are executed by the processor 410, they implement the various processes of the above-described computing engine processing method embodiment and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0117] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0118] Figure 4 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0119] The electronic device 400 includes, but is not limited to, components such as: a radio frequency unit 401, a network module 402, an audio output unit 403, an input unit 404, a sensor 405, a display unit 406, a user input unit 407, an interface unit 408, a memory 409, and a processor 410. The user input unit 407 includes a touch panel 4071 and other input devices 4072; the display unit 406 includes a display panel 4061; and the input unit includes a graphics processor 4041 and a microphone 4042.

[0120] Those skilled in the art will understand that the electronic device 400 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 410 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 4 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0121] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described computing engine processing method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0122] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0123] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described computing engine processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0124] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0125] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0126] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0127] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0128] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A processing method for a computing engine, characterized in that, The computing engine includes multiple different execution engines, each with a corresponding engine identifier, and the method includes: Data acquisition and computation tasks; The initial task execution plan for the data computing task is determined by a pre-built intelligent decision-making model. The data computation task is verified using the priority configuration rules in the intelligent decision-making model, and the initial task execution plan is adjusted based on the verification results to obtain the target task execution plan; wherein, the target task execution plan includes the target engine identifier and the target resource configuration parameters; Based on the target resource configuration parameters, the target execution engine corresponding to the target engine identifier is used to process the data calculation task.

2. The method according to claim 1, characterized in that, The intelligent decision-making model includes an environmental perception module. The step of determining the initial task execution plan for the data computation task using the pre-built intelligent decision-making model includes: Based on the data calculation task, determine the first input feature for inputting into the intelligent decision-making model; The environmental perception module obtains second input features for inputting into the intelligent decision-making model. Based on the first input feature and the second input feature, an initial task execution plan for the data computation task is determined.

3. The method according to claim 2, characterized in that, The first input features include the target SQL operator and / or combination of target SQL operators corresponding to the data calculation task and the estimated data volume; The second input feature includes hardware information and resource usage information.

4. The method according to any one of claims 1-3, characterized in that, The priority configuration rule includes multiple preset verification conditions. Adjusting the initial task execution plan based on the verification results to obtain the target task execution plan includes: If the data calculation task does not meet the multiple preset verification conditions, the initial task execution plan will be used as the target task execution plan. If the data calculation task meets at least one preset verification condition, the initial task execution plan is adjusted according to the priority configuration rule to obtain the target task execution plan.

5. The method according to claim 4, characterized in that, The priority configuration rule includes at least one first preset verification condition, at least one second preset verification condition, and at least one third preset verification condition. The first preset verification condition is used to verify the task characteristics of the data calculation task, the second preset verification condition is used to verify the target SQL operator of the data calculation task, and the third preset verification condition is used to verify the estimated data volume of the data calculation task. When the data calculation task meets the at least one preset verification condition, the initial task execution plan is adjusted according to the priority configuration rule to obtain the target task execution plan, including: If the data calculation task meets the first preset verification condition, the second engine identifier is determined based on the priority configuration rule; If the data calculation task meets the second preset verification condition, the second engine identifier is determined based on the priority configuration rule; If the data calculation task meets the third preset verification condition, the second resource configuration parameters are determined based on the priority configuration rules. The initial task execution plan is adjusted using the second engine identifier and / or the second resource configuration parameters to obtain the target task execution plan.

6. The method according to claim 5, characterized in that, The initial task execution plan includes a first engine identifier and a first resource configuration parameter. Adjusting the initial task execution plan using the second engine identifier and / or the second resource configuration parameter to obtain the target task execution plan includes: If the first engine identifier and the second engine identifier are inconsistent, the first engine identifier shall be replaced with the second engine identifier; If the first resource configuration parameter and the second resource configuration parameter are inconsistent, the first resource configuration parameter shall be replaced with the second resource configuration parameter; The adjusted initial task execution plan will be used as the target task execution plan.

7. The method according to claim 1, characterized in that, The intelligent decision-making model is constructed in the following manner: Obtain historical task information, as well as corresponding historical execution feedback information and historical resource usage information; Based on the historical task information and the historical execution feedback information, a computing engine analysis library is constructed; Based on the historical task information and the historical resource usage information, a configuration parameter analysis library is constructed. The intelligent decision-making model is obtained based on the computing engine analysis library and the configuration parameter analysis library.

8. The method according to claim 7, characterized in that, The step of constructing a computing engine analysis library based on the historical task information and the historical execution feedback information includes: Obtain each historical SQL operator and / or combination of historical SQL operators from the historical task information; Based on the historical execution feedback information, determine the recommendation engine identifier corresponding to each historical SQL operator and / or combination of historical SQL operators; The computational engine analysis library is constructed based on the various historical SQL operators and / or combinations of historical SQL operators and the corresponding recommendation engine identifiers.

9. The method according to claim 7, characterized in that, The step of constructing a configuration parameter analysis library based on the historical task information and the historical resource usage information includes: Obtain the combinations of historical SQL operators from the historical task information; Based on the historical resource usage information, the recommended configuration parameters corresponding to each historical SQL operator combination are determined; Based on the various historical SQL operator combinations and their corresponding recommended configuration parameters, a configuration parameter analysis library is constructed.

10. A processing apparatus for a computing engine, characterized in that, The computing engine includes multiple different execution engines, each execution engine having a corresponding engine identifier; the device includes: The task acquisition module is used to acquire data calculation tasks; The plan generation module is used to determine the initial task execution plan for the data computing task through a pre-built intelligent decision-making model. The plan optimization module is used to verify the data calculation task through the priority configuration rules in the intelligent decision model, and adjust the initial task execution plan based on the verification results to obtain the target task execution plan; wherein, the target task execution plan includes the target engine identifier and the target resource configuration parameters; The task processing module is used to process the data calculation task based on the target resource configuration parameters and the target execution engine corresponding to the target engine identifier.

Images