Big data task resource optimization method and device and electronic equipment

Abstract

Embodiments of the present application provide a big data task resource optimization method, device and electronic equipment, relating to the technical field of big data. The method receives task information to be optimized which explicitly contains a target module identifier and a target data volume, combines a performance baseline table which is pre-constructed and generated based on real historical running data of each module, accurately determines a target performance baseline and a target performance constraint, and accordingly performs closed-loop optimization on an initial resource configuration amount, and finally outputs a target resource configuration amount which meets performance requirements. In this process, performance constraints are taken as rigid targets, and reverse calculation and dynamic adaptation are achieved through the linkage of the target data volume, the target performance baseline and the target performance constraint, which not only avoids reliance on artificial experience and blind allocation of resources, but also covers different scales of data scenarios. Meanwhile, since the performance baseline table is independently constructed according to modules, it naturally supports differentiated optimization of modules, and significantly improves the compliance, automation and system adaptation capability of resource configuration.

Description

Technical Field

[0001] This invention relates to the field of big data technology, and more specifically, to a method, apparatus, and electronic device for optimizing big data task resources. Background Technology

[0002] In big data processing scenarios, key runtime configuration parameters of distributed computing frameworks (such as Apache Spark) have a decisive impact on the overall execution efficiency of jobs, cluster resource utilization, and system stability.

[0003] Current mainstream parameter tuning methods are still mainly driven by human experience, which has significant limitations: on the one hand, it is difficult to dynamically adapt to drastic fluctuations in input data volume (such as jumping from hundreds of GB to tens of thousands of GB), resulting in serious configuration lag; on the other hand, it ignores the inherent differences in computing characteristics of different business modules (such as IO-intensive, CPU-intensive, or memory-intensive), which can easily lead to resource mismatch. Therefore, it is urgent to build an intelligent parameter optimization mechanism with perception capabilities, modelability, and generalization to support the highly reliable, efficient, and elastic operation of large-scale data processing systems. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a method, apparatus and electronic device for optimizing big data task resources, which can improve the compliance of resource allocation.

[0005] To achieve the above objectives, the technical solutions adopted in the embodiments of the present invention are as follows: In a first aspect, the present invention provides a method for optimizing resources for big data tasks, the method comprising: Receive information about the task to be optimized; the information about the task to be optimized includes the target module identifier and the target data volume. The target performance baseline, target performance constraints, and initial resource allocation are determined based on the target module identifier, the target data volume, and a pre-built performance baseline table; the performance baseline table is constructed based on the historical operating data and resource configuration of each module. The initial resource allocation is optimized based on the target data volume, the target performance baseline, and the target performance constraints to obtain the target resource allocation.

[0006] In an optional implementation, determining the target performance baseline, target performance constraints, and initial resource allocation based on the target module identifier, the target data volume, and a pre-built performance baseline table includes: If the target module identifier exists in the performance baseline table, the performance baseline and performance constraint corresponding to the target module identifier in the performance baseline table are determined as the target performance baseline and the target performance constraint. The initial resource allocation is obtained by using a pre-trained resource prediction model based on the target module identifier, the target data volume, and the target performance baseline. If the target module identifier is not found in the performance baseline table, the default performance baseline, default performance constraint, and default resource configuration amount are determined as the target performance baseline, the target performance constraint, and the initial resource configuration amount, respectively.

[0007] In an optional implementation, the target performance baseline includes the average garbage collection rate corresponding to the target module identifier. The step of using a pre-trained resource prediction model to perform resource prediction based on the target module identifier, the target data volume, and the target performance baseline to obtain the initial resource allocation includes: The target module identifier, the target data volume, and the average garbage collection ratio corresponding to the target module identifier are input into a pre-trained resource prediction model to perform resource prediction and obtain the initial resource allocation.

[0008] In an optional implementation, the step of optimizing the initial resource allocation based on the target data volume, the target performance baseline, and the target performance constraints to obtain the target resource allocation includes: Use the preset adjustment coefficient as the target adjustment coefficient; The initial resource allocation is optimized based on the target adjustment coefficient, the target data volume, the target performance constraint, and the target performance baseline to obtain the optimized resource allocation. If the optimized resource allocation meets the target performance constraint, the optimized resource allocation is determined as the target resource allocation. If the optimized resource allocation does not meet the target performance constraint, a new target adjustment coefficient is determined based on the target adjustment coefficient and the preset increase, and optimization is performed using the new target adjustment coefficient. If the new target adjustment coefficient reaches the adjustment threshold and the optimized resource allocation does not meet the target performance constraint, the optimized resource allocation corresponding to the new target adjustment coefficient is determined as the target resource allocation, and the target resource allocation is marked as to be optimized.

[0009] In an optional implementation, the target performance baseline includes the time statistics threshold, unit data consumption time, average garbage collection ratio, and average memory utilization rate corresponding to the target module identifier; the step of optimizing the initial resource allocation based on the target adjustment coefficient, the target data volume, the target performance constraints, and the target performance baseline to obtain the optimized resource allocation includes: The target time threshold is determined based on the time statistics threshold corresponding to the target module identifier and the target adjustment coefficient; The initial resource allocation is optimized based on the target time consumption threshold, the target data volume, the target performance constraints, and the unit data consumption time, average garbage collection ratio, and average memory utilization rate corresponding to the target module identifier, to obtain the optimized resource allocation.

[0010] In an optional implementation, the initial resource configuration includes the initial number of executor instances, the initial executor memory capacity, the initial number of executor cores, and the initial number of partitions; the target performance constraint includes a garbage collection percentage constraint; the optimization of the initial resource configuration based on the target time consumption threshold, the target data volume, the target performance constraint, and the unit data consumption time, average garbage collection percentage, and average memory utilization rate corresponding to the target module identifier, to obtain the optimized resource configuration, includes: The optimized number of actuator instances is determined based on the initial number of actuator instances, the unit data consumption time, the target data volume, the target consumption time threshold, and the preset maximum number of actuators. Based on the average garbage collection ratio and the garbage collection ratio constraint, the initial capacity of the actuator memory is optimized to obtain the optimized actuator memory capacity. The initial number of partitions is optimized based on the optimized number of executor instances, the preset low threshold for executors, and the preset high threshold for executors, resulting in the optimized number of partitions. The optimized resource allocation is composed of the optimized number of executor instances, the optimized executor memory capacity, the optimized number of partitions, and the initial number of executors cores.

[0011] In an optional implementation, the step of optimizing the initial capacity of the actuator memory based on the average garbage collection ratio and the garbage collection ratio constraint to obtain the optimized actuator memory capacity includes: If the average garbage collection ratio meets the garbage collection ratio constraint, the initial memory capacity of the actuator will be updated to the first memory specification; If the average garbage collection ratio does not meet the garbage collection ratio constraint, the initial memory capacity of the actuator is updated to the second memory specification; the first memory specification is smaller than the second memory specification.

[0012] In an optional implementation, the step of optimizing the initial number of partitions based on the optimized number of executor instances, a preset low threshold for executors, and a preset high threshold for executors to obtain the optimized number of partitions includes: If the number of executor instances after optimization is less than the preset low threshold for executors, the initial number of partitions will be updated to the first partition specification. If the number of optimized executor instances is not less than the preset low threshold for executors and the number of optimized executor instances is less than the preset high threshold for executors, then the initial number of partitions will be updated to the second partition specification. If the number of executor instances after optimization is not less than the preset high threshold for executors, the initial number of partitions is updated to the third partition specification; the first partition specification is less than the second partition specification, and the second partition specification is less than the third partition specification.

[0013] Secondly, the present invention provides a big data task resource optimization device, the device comprising: The acquisition module is used to receive information about the task to be optimized; the information about the task to be optimized includes the target module identifier and the target data volume. The processing module is used to determine the target performance baseline, target performance constraints, and initial resource allocation based on the target module identifier, the target data volume, and a pre-built performance baseline table; the performance baseline table is constructed based on the historical operating data and resource configuration of each module. The optimization module is used to optimize the initial resource allocation based on the target data volume, the target performance baseline, and the target performance constraints to obtain the target resource allocation.

[0014] Thirdly, the present invention provides an electronic device including a processor and a memory, wherein the memory stores a computer program executable by the processor, and the processor can execute the computer program to implement the big data task resource optimization method described in any of the foregoing embodiments.

[0015] Compared to existing technologies, the big data task resource optimization method, apparatus, and electronic device provided in this invention receive task information that explicitly includes the target module identifier and target data volume. Combined with a pre-built performance baseline table generated based on real historical operating data of each module, the method accurately determines the target performance baseline and target performance constraints. Based on this, it performs closed-loop optimization of the initial resource allocation, ultimately outputting the target resource allocation that meets performance requirements. This process uses performance constraints as a rigid target, achieving reverse calculation and dynamic adaptation through the linkage of the target data volume, target performance baseline, and target performance constraints. This avoids reliance on manual experience and blind resource allocation, and can cover data scenarios ranging from hundreds of GB to tens of thousands of GB. Furthermore, because the performance baseline table is built independently for each module, it naturally supports differentiated optimization by module, significantly improving the compliance, automation, and system adaptability of resource allocation.

[0016] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This illustration shows a flowchart of a big data task resource optimization method provided by an embodiment of the present invention.

[0019] Figure 2 This illustration shows another flowchart of the big data task resource optimization method provided in an embodiment of the present invention.

[0020] Figure 3 This diagram illustrates a block diagram of a big data task resource optimization device provided in an embodiment of the present invention.

[0021] Figure 4 A block diagram of an electronic device provided in an embodiment of the present invention is shown. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0023] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0024] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0025] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0026] Please refer to Figure 1 , Figure 1 This diagram illustrates a flowchart of a big data task resource optimization method provided by an embodiment of the present invention. The method includes the following steps: Step S10: Receive the task information to be optimized; the task information to be optimized includes the target module identifier and the target data volume.

[0027] In this embodiment of the invention, a task information to be optimized is received, which includes, but is not limited to, the target module identifier (e.g., module2) and the target data volume (e.g., 4000GB). This is equivalent to receiving a clear instruction: generate a suitable set of resource configurations for the task of module2 with 4000GB of data. The entire process is thus initiated without relying on any human intervention.

[0028] Step S20: Determine the target performance baseline, target performance constraints, and initial resource configuration based on the target module identifier, target data volume, and pre-built performance baseline table; the performance baseline table is constructed based on the historical operating data and resource configuration of each module.

[0029] In this embodiment of the invention, the target performance baseline, target performance constraints, and initial resource allocation are obtained by combining the target module identifier, target data volume, and a pre-built performance baseline table. The performance baseline table is a structured knowledge base formed by statistical analysis of real-world operational data accumulated over a long period by each business module. The performance baseline table includes record items for each business module, and each record item includes the performance baseline and performance constraints. The real-world operational data includes the amount of input data for each run, total task time, garbage collection time, average memory usage, and historical resource configuration parameters (including but not limited to the number of executor instances, memory size, number of cores, and number of Shuffle partitions).

[0030] Taking module2 as an example, based on 200 historical runs, the historical average run time is 50 minutes, and the historical average input size is 1000GB. Therefore, the time consumed per unit of data is calculated to be 50 / 1000 = 0.05 minutes / GB. The 95th percentile of the historical run times is calculated, resulting in a time threshold of 60 minutes. The preset adjustment coefficient is 1.2, the preset increase is 0.1, and the adjustment threshold is 2. The total garbage collection time across the 200 runs is 60 seconds, and the total task time is 600 seconds. Therefore, the average garbage collection percentage is calculated to be 0.1%.

[0031] Performance constraints are set based on the constraints of module2 and the resource utilization characteristics of the Spark framework: the runtime does not exceed the target time threshold; the average garbage collection time percentage does not exceed the garbage collection time percentage threshold of 0.2; and the average memory utilization rate is within the reasonable range of memory utilization [0.6,1].

[0032] Step S30: Optimize the initial resource allocation based on the target data volume, target performance baseline, and target performance constraints to obtain the target resource allocation.

[0033] In this embodiment of the invention, the target data volume, target performance baseline, and target performance constraints are combined to perform reverse optimization of the initial resource allocation, and finally output a set of target resource allocation that avoids resource waste and can effectively ensure the timely and stable completion of tasks.

[0034] In summary, the big data task resource optimization method provided by this invention receives task information that explicitly includes the target module identifier and target data volume. Combined with a pre-built performance baseline table generated based on real historical operating data of each module, it accurately determines the target performance baseline and target performance constraints. Based on this, it performs closed-loop optimization of the initial resource allocation, ultimately outputting the target resource allocation that meets performance requirements. This process uses performance constraints as a rigid target, achieving reverse calculation and dynamic adaptation through the linkage of the target data volume, target performance baseline, and target performance constraints. This avoids reliance on manual experience and blind resource allocation, and can cover data scenarios ranging from hundreds of GB to tens of thousands of GB. Furthermore, because the performance baseline table is built independently for each module, it naturally supports differentiated optimization by module, significantly improving the compliance, automation, and system adaptability of resource allocation.

[0035] Optionally, regarding how to determine the target performance baseline, target performance constraints, and initial resource allocation for the target module, the following is a possible implementation method. Figure 1 The sub-steps of step S20 may include: Step S200: If a target module identifier exists in the performance baseline table, the performance baseline and performance constraints corresponding to the target module identifier in the performance baseline table are determined as the target performance baseline and target performance constraints.

[0036] In this embodiment of the invention, a performance baseline table is searched based on the received target module identifier. If the performance baseline table records a performance baseline and performance constraint that match the target module identifier, the matching performance baseline is determined as the target performance baseline, and the matching performance constraint is determined as the target performance constraint.

[0037] For example, if the target module is module2, the performance baseline table records performance baselines for module2 based on statistical analysis of multiple historical runtime data, including but not limited to average garbage collection percentage, unit data consumption time, and average memory utilization. Utilizing the module's historical performance guides resource allocation, improving the accuracy and reliability of the baseline.

[0038] Step S210: Use a pre-trained resource prediction model to predict resources based on the target module identifier, target data volume, and target performance baseline to obtain the initial resource allocation.

[0039] Assuming that the target module identifier exists in the performance baseline table, the target module identifier, the target data volume, and the newly determined target performance baseline are input into a pre-trained resource prediction model, which then outputs the initial resource allocation.

[0040] It should be noted that the resource prediction model is trained based on historical operating data. The input features include the module identifier (after one-hot encoding), data volume, and the performance baseline of the corresponding module. The output directly corresponds to the initial resource configuration. The initial resource configuration includes, but is not limited to, the initial number of executor instances, the initial executor memory capacity, the initial number of executor cores, and the initial number of partitions.

[0041] For example, for module 2 and a target data volume of 4000GB, the resource prediction model outputs the following initial values: executor_instances_init=270, executor_memory_init=20GB, executor_cores_init=4, and shuffle_partitions_init=3500. These predicted values ​​are not manually set empirical values, but rather statistical patterns learned by the resource prediction model from massive historical tasks, demonstrating data-driven and generalization capabilities.

[0042] Step S220: If the target module identifier is not found in the performance baseline table, the default performance baseline, default performance constraint, and default resource configuration amount are determined as the target performance baseline, target performance constraint, and initial resource configuration amount, respectively.

[0043] In this embodiment of the invention, if no target module identifier is found in the performance baseline table, such as the newly launched module4 having no running records, the preset default configuration strategy is automatically enabled, the default performance baseline is determined as the target performance baseline, the default performance constraint is determined as the target performance constraint, and the default resource configuration amount is determined as the initial resource configuration amount.

[0044] As one possible implementation, for example, based on empirical values ​​derived from historical operational data of existing modules of the same type, the default unit data consumption time can be set to 0.05 minutes per GB. The time statistics threshold can be set to 30 minutes, and the preset adjustment coefficient can be set to 1.5. The default performance constraints are consistent with the performance constraints of the same type of module in the performance baseline table, i.e., the average garbage collection percentage does not exceed the garbage collection percentage threshold of 0.2, and the reasonable range of memory utilization is between 60% and 100%. Accordingly, the default resource configuration can be set as follows: the initial number of executor instances is set to 80, the initial executor memory capacity is set to 25GB, the initial number of executor cores is set to 4, and the initial number of partitions is set to 4096. The default parameters have been verified to be feasible in similar scenarios, ensuring basic task startup and stable operation.

[0045] As can be seen, by distinguishing whether the target module has historical running data, the embodiments of the present invention can flexibly determine the benchmark and initial configuration required for subsequent optimization. This not only ensures the personalized adaptation capability of modules with data, but also solves the practical problem that modules without historical data cannot start optimization, significantly improving the applicability of the big data task resource optimization method.

[0046] Optionally, the target performance baseline includes the average garbage collection percentage corresponding to the target module identifier. Regarding how to use a resource prediction model to predict the initial resource allocation corresponding to the target module, a possible implementation method is provided below. The sub-steps of step S210 may include: Input the target module identifier, target data volume, and average garbage collection ratio corresponding to the target module identifier into a pre-trained resource prediction model to perform resource prediction and obtain the initial resource allocation.

[0047] In this embodiment of the invention, after extracting the average garbage collection ratio corresponding to the target module identifier, the target module identifier, the target data volume, and the average garbage collection ratio corresponding to the target module identifier are used together as input data for the resource prediction model. The resource prediction model is used to perform resource prediction, and the output of the resource prediction model is the initial resource configuration amount such as the initial number of executor instances and the initial capacity of executor memory.

[0048] The resource prediction model is a regression model trained on historical operational data of each module, such as a random forest regression model. It can also be replaced by other predictive models, such as neural networks. The prediction results output by the trained resource prediction model can reflect the individual characteristics of each module while taking into account the data scale and memory health, thus improving the rationality of the initial resource allocation.

[0049] Optionally, regarding how to determine the target performance baseline, target performance constraints, and initial resource allocation for the target module, a possible implementation method is provided below. Please refer to... Figure 2 , Figure 1 The sub-steps of step S30 may include: Step S300: Use the preset adjustment coefficient as the target adjustment coefficient.

[0050] In this embodiment of the invention, when initially optimizing the initial resource configuration, a preset adjustment coefficient is used as the initial target adjustment coefficient. It should be understood that the preset adjustment coefficient is not fixed and can be flexibly set according to the importance of the module. For example, the preset adjustment coefficient for core modules can be set to 1.2, and the preset adjustment coefficient for non-core modules can be set to 1.5. The preset adjustment coefficient can be set according to the differentiated requirements for response speed in different business scenarios, and this invention does not limit this.

[0051] Step S310: Optimize the initial resource allocation based on the target adjustment coefficient, target data volume, target performance constraints, and target performance baseline to obtain the optimized resource allocation.

[0052] Next, based on the target adjustment coefficient, target data volume, target performance constraints, and target performance baselines such as the time statistics threshold, unit data consumption time, average garbage collection ratio, and average memory utilization rate corresponding to the target module identifier, the initial resource allocation is co-optimized to obtain the optimized resource allocation.

[0053] Step S320: If the optimized resource allocation meets the target performance constraints, the optimized resource allocation is determined as the target resource allocation.

[0054] Subsequently, a performance compliance check is performed on the optimized resource allocation. This involves verifying whether each optimized resource allocation meets the preset performance constraints (i.e., target performance constraints), including but not limited to whether the runtime of the target module (i.e., the product of the time consumed per unit of data and the target data volume) does not exceed the target runtime threshold, whether the average garbage collection percentage does not exceed the garbage collection percentage threshold, and whether the average memory utilization does not exceed the reasonable range of memory utilization. If all are satisfied, the optimized resource allocation is directly determined as the target resource allocation.

[0055] Step S330: If the optimized resource allocation does not meet the target performance constraints, determine a new target adjustment coefficient based on the target adjustment coefficient and the preset increase, and use the new target adjustment coefficient for optimization.

[0056] If any condition is not met, the optimized resource allocation is deemed not to meet the target performance constraints, and the optimization fails to meet expectations, thus entering the iteration process. The sum of the current target optimization coefficient and the preset increase (e.g., an increase of 0.1 each time) is determined as the new target optimization coefficient, which is used for the next optimization.

[0057] For example, the product of the time consumed per unit of data and the target data volume is: minutes, meaning the running time is greater than the target time threshold (e.g.) The runtime constraint is not met. The average memory utilization rate is 0.2, which is below the lower limit of the reasonable range for memory utilization (e.g., [0.6, 1.0]), and the memory constraint is not met. The average garbage collection percentage is 0.1, which is below the garbage collection percentage threshold, and the garbage collection percentage constraint is met. Since the target performance constraints are not fully met, the iteration process begins. For example, the target adjustment factor is increased from 1.2 to 1.3, and the target time consumption threshold is recalculated based on the new target adjustment factor and the time statistics threshold corresponding to the target module identifier. The initial resource configuration is then optimized again based on the target time consumption threshold, target data volume, target performance constraints, and the unit data consumption time, average garbage collection ratio, and average memory utilization corresponding to the target module identifier. Finally, it is determined whether the optimized resource configuration meets the target performance constraints.

[0058] Step S340: If the new target adjustment coefficient reaches the adjustment threshold and the optimized resource allocation does not meet the target performance constraints, the optimized resource allocation corresponding to the new target adjustment coefficient is determined as the target resource allocation, and the target resource allocation is marked as to be optimized.

[0059] In this embodiment of the invention, steps S310-S330 and their corresponding sub-steps are continuously executed until the optimized resource allocation meets the target performance constraints, or the new target adjustment coefficient reaches a preset adjustment threshold (e.g., 2.0). If the target performance constraints are still not met after the new target adjustment coefficient reaches the adjustment threshold, optimization is no longer forced. Instead, the currently optimized resource allocation is determined as the target resource allocation and simultaneously marked as "to be optimized" so that operations and maintenance personnel can retrain the resource prediction model to further improve the accuracy of the resource prediction model for resource prediction.

[0060] As can be seen, the embodiments of the present invention initiate optimization by setting a preset adjustment coefficient. Based on the target data volume, target performance baseline, and target performance constraints, the initial resource allocation is calculated for the first time to obtain the optimized resource allocation. If all target performance constraints are met, it is determined as the target resource allocation. If not, the adjustment coefficient is increased by a preset increment and the process is retried. When the adjustment coefficient reaches a preset adjustment threshold and still does not meet the requirements, the current optimization result is determined as the target resource allocation and marked as "to be optimized". This not only improves the stability of task operation through multiple rounds of constraint verification, but also provides identifiable and traceable prompts when the target cannot be automatically met, ensuring that the entire optimization process always has results, evidence, and an exit, while taking into account both automation efficiency and engineering controllability.

[0061] Optionally, the target performance baseline includes the time statistics threshold, unit data consumption time, average garbage collection rate, and average memory utilization rate corresponding to the target module identifier. Regarding resource optimization, one possible implementation method is provided below. Figure 2 The sub-steps of step S310 may include: Step S311: Determine the target time threshold based on the time statistics threshold and target adjustment coefficient corresponding to the target module identifier.

[0062] In this embodiment of the invention, the time statistics threshold is determined based on the historical running time of the target module. If the target module does not have historical running data, it can be preset based on the historical data of similar modules. For example, for the core business module "module2", the 95th percentile (e.g., 60 minutes) of the historical 200 running times is determined as the time statistics threshold for module2, representing the completion time level that module2 can achieve in most cases (95% probability), and has strong stability and reference value.

[0063] The target time threshold is obtained by multiplying the time statistics threshold corresponding to the target module identifier by the target adjustment coefficient used in the current optimization iteration. For example, if the target adjustment coefficient used in the current optimization iteration is 1.2, the calculated time threshold is... Minutes, meaning the target time threshold is 72 minutes.

[0064] Step S312: Optimize the initial resource configuration based on the target time threshold, target data volume, target performance constraints, and the unit data time, average garbage collection ratio, and average memory utilization rate corresponding to the target module identifier, to obtain the optimized resource configuration.

[0065] In this embodiment of the invention, based on the target time consumption threshold and combined with performance baseline indicators such as target data volume, unit data consumption time, average garbage collection ratio and average memory utilization, the initial resource configuration is optimized as a whole to ensure that the final output resource configuration can meet performance constraints and has engineering feasibility.

[0066] It should be noted that the time statistics threshold is not fixed at a specific quantile, but can be flexibly selected based on the stability requirements of the actual business. For example, in core business scenarios, if the configuration scheme is to cover the vast majority of operating conditions, the time statistics threshold can be set to the 95th quantile of historical running time; if the business has extremely high requirements for response speed and a small fault tolerance margin, a more conservative 99th quantile can be used; conversely, if the task data fluctuates greatly and a certain proportion of occasional timeouts is acceptable, the 90th quantile can be used to improve resource utilization efficiency. The specific quantile used depends on the actual deployment needs, and this invention does not limit it.

[0067] Optionally, the initial resource configuration includes, but is not limited to, the initial number of executor instances, the initial executor memory capacity, the initial number of executor cores, and the initial number of partitions. Target performance constraints include, but are not limited to, garbage collection percentage constraints. Regarding how to optimize the initial number of executor instances, the initial executor memory capacity, the initial number of executor cores, and the initial number of partitions, a possible implementation is provided below. The sub-steps of step S312 may include: Step S312-1: Determine the optimized number of actuator instances based on the initial number of actuator instances, unit data consumption time, target data volume, target consumption time threshold, and preset maximum number of actuators.

[0068] In this embodiment of the invention, the initial number of actuator instances is adjusted in reverse according to the correlation between unit data consumption time, target data volume and target consumption time threshold, and boundary truncation is performed in combination with the preset maximum number of actuators, thereby determining the optimized number of actuator instances.

[0069] As one possible implementation, suppose the tuning formula for the number of executor instances is set as follows:

[0070] in, This is the number of executor instances after optimization; This is the initial number of executor instances; This is the maximum preset number of actuators; It is the time consumed per unit of data; It is the target data volume; `min()` is the target time threshold; `min()` is the function to take the minimum value; `round()` is the rounding function.

[0071] For example, suppose the Spark-based cluster resources have a preset minimum number of executors of 10 and a preset maximum number of executors of 200. For instance, the initial number of executor instances is 270, the unit data processing time is 0.05 minutes / gigabyte, the target data volume is 4000 gigabytes, and the target processing time threshold is 72 minutes. Using the above formula for optimizing the number of executor instances, we can work backwards to calculate that the optimized number of executor instances is 200, ensuring that it does not exceed the cluster resource capacity.

[0072] It should be understood that, under the premise of a limited total time consumption, the optimized number of executor instances is directly proportional to the total workload (i.e., the target data volume) and inversely proportional to the data processing capacity (i.e., directly proportional to the time consumed per unit of data). In practical applications, the optimization formula for the number of executor instances is not unique. Other calculation formulas that can reflect the same inverse constraint relationship can be used according to the actual application scenario, and this invention does not limit this.

[0073] Step S312-2: Based on the average garbage collection ratio and the garbage collection ratio constraint, the initial capacity of the actuator memory is optimized to obtain the optimized actuator memory capacity.

[0074] In this embodiment of the invention, the average garbage collection ratio corresponding to the target module identifier is read and compared with the preset garbage collection ratio constraint. The initial capacity of the executor memory is optimized based on the comparison result, thereby achieving dynamic matching between memory configuration and actual operating load.

[0075] Step S312-3: Optimize the initial number of partitions based on the optimized number of executor instances, the preset low threshold for executors, and the preset high threshold for executors to obtain the optimized number of partitions; the optimized resource allocation consists of the optimized number of executor instances, the optimized executor memory capacity, the optimized number of partitions, and the initial number of executors cores.

[0076] In this embodiment of the invention, the initial number of partitions is the predicted number of logical partitions used for data redistribution during the Shuffle phase in Spark. The adjustment of the initial number of partitions is not an independent decision, but rather follows the determined number of executor instances.

[0077] Finally, the optimized resource configuration consists of the number of executor instances, the optimized executor memory capacity, the optimized number of partitions, and the initial number of executor cores (based on best practices of the Spark framework, such as 4). This forms a set of resource optimization results that are mutually compatible and controllable, avoiding both insufficient computing power or wasted memory, and preventing system bottlenecks caused by mismatched parallelism.

[0078] It should be noted that in some implementations, the initial number of cores for the executor may not be a fixed value, but rather dynamically adjusted based on other performance metrics. For example, if CPU utilization constraints are also set (such as requiring it to not exceed 80%), the initial number of cores for the executor can be optimized in conjunction with these constraints to obtain the optimized number of executor cores. This optimized number of cores, along with the optimized number of executor instances, the optimized executor memory capacity, and the optimized number of partitions, constitutes the optimized resource allocation. Whether to optimize a particular parameter can be flexibly chosen based on actual business needs, and this invention does not impose any limitations on this.

[0079] Optionally, regarding how to generate the optimized executor memory capacity, the following is a possible implementation. The sub-steps of step S312-2 may include: If the average garbage collection percentage meets the garbage collection percentage constraint, the initial memory capacity of the actuator is updated to the first memory specification; if the average garbage collection percentage does not meet the garbage collection percentage constraint, the initial memory capacity of the actuator is updated to the second memory specification; the first memory specification is smaller than the second memory specification.

[0080] In this embodiment of the invention, the initial capacity of the actuator memory is dynamically adjusted based on whether the historical average garbage collection ratio of the target module meets the preset garbage collection ratio constraint, thereby achieving differentiated adaptation of memory configuration.

[0081] In practical applications, firstly, the average garbage collection percentage and garbage collection percentage constraint corresponding to the target module are obtained. It is assumed that the garbage collection percentage constraint is that the average garbage collection percentage is less than or equal to a garbage collection percentage threshold of 0.2. Next, it is determined whether the average garbage collection percentage meets the requirements of the garbage collection percentage constraint.

[0082] If the average garbage collection percentage of the target module is less than or equal to the garbage collection percentage threshold, then the average garbage collection percentage of the target module meets the garbage collection percentage constraint, indicating that the memory pressure of the target module was relatively low during historical operation. The initial memory capacity of the executor is then updated to the first memory specification, such as the standard memory specification of 12GB, to avoid wasting memory resources.

[0083] If the average garbage collection percentage of the target module is greater than the garbage collection percentage threshold, then the average garbage collection percentage of the target module does not meet the garbage collection percentage constraint, indicating that the target module has a risk of frequent garbage collection. The initial memory capacity of the actuator is then updated to the second memory specification, such as the high memory specification of 25GB, to alleviate memory pressure.

[0084] Finally, the updated initial executor memory capacity is determined as the optimized executor memory capacity. It should be noted that the garbage collection percentage constraint, the first memory specification, and the second memory specification can be configured according to the actual application scenario, and this invention does not limit them.

[0085] As can be seen, the embodiments of the present invention dynamically adjust the memory capacity of the actuator by judging whether the average garbage collection ratio of the target module meets the preset garbage collection ratio constraint, so that the memory resource configuration can be adapted to the actual operating characteristics of different modules; when the garbage collection pressure is low, a smaller memory specification is used to avoid resource waste, and when the pressure is high, a larger memory specification is switched to reduce the performance impact caused by frequent garbage collection, thereby improving resource utilization efficiency while ensuring the stable operation of the task.

[0086] Optionally, regarding how to generate the optimized number of partitions, the following is a possible implementation. The sub-steps of step S312-3 may include: The first step is to update the initial number of partitions to the first partition specification if the number of optimized executor instances is less than the preset low threshold for executors.

[0087] In this embodiment of the invention, a partition refers to a Shuffle partition, which is a unit for dividing parallel tasks in the Spark framework. By segmenting the number of Shuffle partitions with the optimized number of executor instances, collaborative adaptation between resource parameters is achieved.

[0088] After obtaining the number of executor instances after optimization, the number of Shuffle partitions of the corresponding specification is automatically selected based on the range in which the number of optimized executor instances falls. If the number of optimized executor instances is less than the preset low threshold for executors (e.g., 100), the initial number of partitions is updated to the first partition specification (e.g., 2048).

[0089] The first partition specification 2048 is suitable for small- to medium-scale computing scenarios, such as when the number of executors does not exceed 99. 2048 can ensure sufficient parallelism without causing additional scheduling overhead and memory pressure due to too many partitions.

[0090] The second step is to update the initial number of partitions to the second partition specification if the number of optimized executor instances is not less than the preset low threshold and the number of optimized executor instances is less than the preset high threshold.

[0091] In this embodiment of the invention, if the number of optimized executor instances is not less than the preset low threshold (e.g., 100) but less than the preset high threshold (e.g., 200), then the initial number of partitions is updated to the second partition specification (e.g., 4096).

[0092] The second partition specification is suitable for large-scale computing scenarios and can be set to a power of 2. It has a natural advantage in the SparkShuffle mechanism and can more efficiently support the concurrent operation of hundreds of executors. Especially when the number of target executors reaches about 200, it can maximize the data processing throughput.

[0093] The third step is to update the initial number of partitions to the third partition specification if the number of executor instances after optimization is not less than the preset high threshold of executors. The first partition specification is less than the second partition specification, and the second partition specification is less than the third partition specification.

[0094] In this embodiment of the invention, if the number of optimized executor instances is not less than a preset high threshold for executors (e.g., 200), the initial number of partitions is updated to the third partition specification (e.g., 8192). The third partition specification is designed for deployment in ultra-large-scale clusters, further improving parallel granularity and mitigating the risk of data skew.

[0095] The fourth step is to determine the initial number of partitions after the update as the number of partitions after optimization.

[0096] It should be noted that the preset actuator low threshold, preset actuator high threshold, first partition specification, second partition specification, and third partition specification can be configured according to the actual application scenario, and this invention does not limit them.

[0097] As can be seen, the embodiments of the present invention optimize the number of executor instances and the number of Shuffle partitions in a coordinated manner, so that the number of Shuffle partitions always matches the actual executor scale. This ensures parallel processing capabilities while avoiding excessive scheduling overhead due to too many partitions, or data skew and resource contention due to too few partitions, thereby improving the stability and resource utilization efficiency of big data tasks.

[0098] Based on the same inventive concept, the basic principle and technical effects of the big data task resource optimization device provided in this embodiment are the same as those in the above embodiments. For the sake of brevity, any parts not mentioned in this embodiment can be referred to the corresponding content in the above embodiments.

[0099] Please refer to Figure 3 , Figure 3 This is a block diagram of a big data task resource optimization device 400 provided in an embodiment of the present invention. The big data task resource optimization device 400 includes an acquisition module 410, a processing module 420, and an optimization module 430.

[0100] The acquisition module 410 is used to receive information about the task to be optimized; the information about the task to be optimized includes the target module identifier and the target data volume.

[0101] Processing module 420 is used to determine the target performance baseline, target performance constraints, and initial resource configuration based on the target module identifier, target data volume, and a pre-built performance baseline table; the performance baseline table is constructed based on the historical operating data and resource configuration of each module.

[0102] The tuning module 430 is used to tune the initial resource configuration based on the target data volume, target performance baseline and target performance constraints to obtain the target resource configuration.

[0103] In summary, the big data task resource optimization device provided in this embodiment of the invention receives task information that explicitly includes the target module identifier and the target data volume. Combined with a pre-built performance baseline table generated based on the real historical operating data of each module, it accurately determines the target performance baseline and target performance constraints. Based on this, it performs closed-loop optimization of the initial resource allocation, ultimately outputting the target resource allocation that meets performance requirements. This process uses performance constraints as a rigid target, achieving reverse calculation and dynamic adaptation through the linkage of the target data volume, target performance baseline, and target performance constraints. This avoids reliance on manual experience and blind resource allocation, and can cover data scenarios ranging from hundreds of GB to tens of thousands of GB. Furthermore, because the performance baseline table is built independently for each module, it naturally supports differentiated optimization of modules, significantly improving the compliance, automation, and system adaptability of resource allocation.

[0104] Optionally, the processing module 420 is specifically configured to: if a target module identifier exists in the performance baseline table, determine the performance baseline and performance constraints corresponding to the target module identifier in the performance baseline table as the target performance baseline and target performance constraints; use a pre-trained resource prediction model to perform resource prediction based on the target module identifier, target data volume, and target performance baseline to obtain the initial resource allocation amount; if a target module identifier does not exist in the performance baseline table, determine the default performance baseline, default performance constraints, and default resource allocation amount as the target performance baseline, target performance constraints, and initial resource allocation amount, respectively.

[0105] Optionally, the target performance baseline includes the average garbage collection percentage corresponding to the target module identifier. The processing module 420 is specifically used to input the target module identifier, the target data volume, and the average garbage collection percentage corresponding to the target module identifier into a pre-trained resource prediction model to perform resource prediction and obtain the initial resource allocation.

[0106] Optionally, the optimization module 430 is specifically used to use a preset adjustment coefficient as a target adjustment coefficient; optimize the initial resource allocation based on the target adjustment coefficient, target data volume, target performance constraints, and target performance baseline to obtain the optimized resource allocation; if the optimized resource allocation meets the target performance constraints, the optimized resource allocation is determined as the target resource allocation.

[0107] If the optimized resource allocation does not meet the target performance constraints, a new target adjustment coefficient is determined based on the target adjustment coefficient and the preset increase, and the new target adjustment coefficient is used for optimization; if the new target adjustment coefficient reaches the adjustment threshold and the optimized resource allocation does not meet the target performance constraints, the optimized resource allocation corresponding to the new target adjustment coefficient is determined as the target resource allocation, and the target resource allocation is marked as to be optimized.

[0108] Optionally, the target performance baseline includes the time statistics threshold, unit data consumption time, average garbage collection ratio, and average memory utilization rate corresponding to the target module identifier. The tuning module 430 is specifically used to determine the target consumption time threshold based on the time statistics threshold corresponding to the target module identifier and the target adjustment coefficient; and to tune the initial resource allocation based on the target consumption time threshold, target data volume, target performance constraints, and the unit data consumption time, average garbage collection ratio, and average memory utilization rate corresponding to the target module identifier, thus obtaining the tuned resource allocation.

[0109] Optionally, the initial resource allocation includes the initial number of executor instances, the initial executor memory capacity, the initial number of executor cores, and the initial number of partitions. The target performance constraints include garbage collection ratio constraints. The tuning module 430 is specifically used to determine the tuned number of executor instances based on the initial number of executor instances, unit data consumption time, target data volume, target consumption time threshold, and the preset maximum number of executors; to tune the initial executor memory capacity based on the average garbage collection ratio and garbage collection ratio constraints; to tune the initial number of partitions based on the tuned number of executor instances, the preset low threshold for executors, and the preset high threshold for executors; and to tune the resource allocation based on the tuned number of executor instances, the tuned executor memory capacity, the tuned number of partitions, and the initial number of executor cores.

[0110] Optionally, the tuning module 430 is specifically used to update the initial capacity of the actuator memory to the first memory specification if the average garbage collection ratio meets the garbage collection ratio constraint; and to update the initial capacity of the actuator memory to the second memory specification if the average garbage collection ratio does not meet the garbage collection ratio constraint; wherein the first memory specification is smaller than the second memory specification.

[0111] Optionally, the optimization module 430 is specifically used to update the initial number of partitions to the first partition specification if the number of optimized executor instances is less than the preset low threshold of executors; update the initial number of partitions to the second partition specification if the number of optimized executor instances is not less than the preset low threshold of executors and the number of optimized executor instances is less than the preset high threshold of executors; and update the initial number of partitions to the third partition specification if the number of optimized executor instances is not less than the preset high threshold of executors. The first partition specification is less than the second partition specification, and the second partition specification is less than the third partition specification.

[0112] Please refer to Figure 4 This is a block diagram illustrating an electronic device 500 provided in an embodiment of the present invention. The electronic device 500 includes, but is not limited to, a personal computer (PC), a personal digital assistant (PDA), a laptop computer, a tablet computer, and a server. The electronic device 500 includes a memory 510, a processor 520, and a communication module 530. The memory 510, processor 520, and communication module 530 are electrically connected directly or indirectly to each other to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0113] The memory 510 is used to store programs or data. The memory 510 may be, but is not limited to, random access memory, read-only memory, programmable read-only memory, erasable read-only memory, electrically erasable read-only memory, etc.

[0114] The processor 520 is used to read / write data or programs stored in the memory 510 and perform corresponding functions. For example, when a computer program stored in the memory 510 is executed by the processor 520, the big data task resource optimization method disclosed in the above embodiments can be implemented.

[0115] The communication module 530 is used to establish a communication connection between the electronic device 500 and other communication terminals via a network, and to send and receive data via the network.

[0116] It should be understood that, Figure 4 The structure shown is only a schematic diagram of the electronic device 500. The electronic device 500 may also include components that are larger than those shown. Figure 4 The more or fewer components shown, or having the same Figure 4 The different configurations shown. Figure 4 The components shown can be implemented using hardware, software, or a combination thereof.

[0117] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor 520, implements the big data task resource optimization method disclosed in the above embodiments.

[0118] This invention also provides a program product that, when executed by processor 520, implements the big data task resource optimization method disclosed in the above embodiments.

[0119] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative; for example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0120] In addition, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0121] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0122] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for optimizing resources for big data tasks, characterized in that, The method includes: Receive information about the task to be optimized; the information about the task to be optimized includes the target module identifier and the target data volume. The target performance baseline, target performance constraints, and initial resource allocation are determined based on the target module identifier, the target data volume, and a pre-built performance baseline table; the performance baseline table is constructed based on the historical operating data and resource configuration of each module. The initial resource allocation is optimized based on the target data volume, the target performance baseline, and the target performance constraints to obtain the target resource allocation.

2. The big data task resource optimization method according to claim 1, characterized in that, The step of determining the target performance baseline, target performance constraints, and initial resource allocation based on the target module identifier, the target data volume, and a pre-built performance baseline table includes: If the target module identifier exists in the performance baseline table, the performance baseline and performance constraint corresponding to the target module identifier in the performance baseline table are determined as the target performance baseline and the target performance constraint. The initial resource allocation is obtained by using a pre-trained resource prediction model based on the target module identifier, the target data volume, and the target performance baseline. If the target module identifier is not found in the performance baseline table, the default performance baseline, default performance constraint, and default resource configuration amount are determined as the target performance baseline, the target performance constraint, and the initial resource configuration amount, respectively.

3. The big data task resource optimization method according to claim 2, characterized in that, The target performance baseline includes the average garbage collection percentage corresponding to the target module identifier. The initial resource allocation is obtained by using a pre-trained resource prediction model based on the target module identifier, the target data volume, and the target performance baseline to perform resource prediction, including: The target module identifier, the target data volume, and the average garbage collection ratio corresponding to the target module identifier are input into a pre-trained resource prediction model to perform resource prediction and obtain the initial resource allocation.

4. The big data task resource optimization method according to claim 1, characterized in that, The step of optimizing the initial resource allocation based on the target data volume, the target performance baseline, and the target performance constraints to obtain the target resource allocation includes: Use the preset adjustment coefficient as the target adjustment coefficient; The initial resource allocation is optimized based on the target adjustment coefficient, the target data volume, the target performance constraint, and the target performance baseline to obtain the optimized resource allocation. If the optimized resource allocation meets the target performance constraint, the optimized resource allocation is determined as the target resource allocation. If the optimized resource allocation does not meet the target performance constraint, a new target adjustment coefficient is determined based on the target adjustment coefficient and the preset increase, and optimization is performed using the new target adjustment coefficient. If the new target adjustment coefficient reaches the adjustment threshold and the optimized resource allocation does not meet the target performance constraint, the optimized resource allocation corresponding to the new target adjustment coefficient is determined as the target resource allocation, and the target resource allocation is marked as to be optimized.

5. The big data task resource optimization method according to claim 4, characterized in that, The target performance baseline includes the time statistics threshold, unit data consumption time, average garbage collection ratio, and average memory utilization rate corresponding to the target module identifier; the step of optimizing the initial resource allocation based on the target adjustment coefficient, the target data volume, the target performance constraints, and the target performance baseline to obtain the optimized resource allocation includes: The target time threshold is determined based on the time statistics threshold corresponding to the target module identifier and the target adjustment coefficient; The initial resource allocation is optimized based on the target time consumption threshold, the target data volume, the target performance constraints, and the unit data consumption time, average garbage collection ratio, and average memory utilization rate corresponding to the target module identifier, to obtain the optimized resource allocation.

6. The big data task resource optimization method according to claim 5, characterized in that, The initial resource configuration includes the initial number of executor instances, the initial executor memory capacity, the initial number of executor cores, and the initial number of partitions. The target performance constraint includes a garbage collection percentage constraint. The initial resource configuration is then optimized based on the target time consumption threshold, the target data volume, the target performance constraint, and the unit data consumption time, average garbage collection percentage, and average memory utilization corresponding to the target module identifier, resulting in the optimized resource configuration. The optimized number of actuator instances is determined based on the initial number of actuator instances, the unit data consumption time, the target data volume, the target consumption time threshold, and the preset maximum number of actuators. Based on the average garbage collection ratio and the garbage collection ratio constraint, the initial capacity of the actuator memory is optimized to obtain the optimized actuator memory capacity. The initial number of partitions is optimized based on the optimized number of executor instances, the preset low threshold for executors, and the preset high threshold for executors, resulting in the optimized number of partitions. The optimized resource allocation is composed of the optimized number of executor instances, the optimized executor memory capacity, the optimized number of partitions, and the initial number of executors cores.

7. The big data task resource optimization method according to claim 6, characterized in that, The step of optimizing the initial capacity of the actuator memory based on the average garbage collection ratio and the garbage collection ratio constraint to obtain the optimized actuator memory capacity includes: If the average garbage collection ratio meets the garbage collection ratio constraint, the initial memory capacity of the actuator will be updated to the first memory specification; If the average garbage collection ratio does not meet the garbage collection ratio constraint, the initial memory capacity of the actuator is updated to the second memory specification; the first memory specification is smaller than the second memory specification.

8. The big data task resource optimization method according to claim 6, characterized in that, The process of optimizing the initial number of partitions based on the optimized number of executor instances, a preset low threshold for executors, and a preset high threshold for executors to obtain the optimized number of partitions includes: If the number of executor instances after optimization is less than the preset low threshold for executors, the initial number of partitions will be updated to the first partition specification. If the number of optimized executor instances is not less than the preset low threshold for executors and the number of optimized executor instances is less than the preset high threshold for executors, then the initial number of partitions will be updated to the second partition specification. If the number of executor instances after optimization is not less than the preset high threshold for executors, the initial number of partitions is updated to the third partition specification; the first partition specification is less than the second partition specification, and the second partition specification is less than the third partition specification.

9. A big data task resource optimization device, characterized in that, The device includes: The acquisition module is used to receive information about the task to be optimized; the information about the task to be optimized includes the target module identifier and the target data volume. The processing module is used to determine the target performance baseline, target performance constraints, and initial resource allocation based on the target module identifier, the target data volume, and a pre-built performance baseline table; the performance baseline table is constructed based on the historical operating data and resource configuration of each module. The optimization module is used to optimize the initial resource allocation based on the target data volume, the target performance baseline, and the target performance constraints to obtain the target resource allocation.

10. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a computer program that can be executed by the processor, the processor being able to execute the computer program to implement the big data task resource optimization method according to any one of claims 1-8.

Images