A computing resource granularity division method in a data lake environment
Patent Information
- Application Number
- CN202611173665.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-04
- Publication Date
- 2026-10-27
AI Technical Summary
[0004]然而，在数据湖全链路作业场景下，为Flink算子设置最优并行度、实现合理的计算资源粒度划分，面临远超通用分布式计算场景的专属挑战
通过结合采集到的集群资源利用率信息，将Flink迭代任务的计算逻辑建模为一个随时间变化的作业图（JobGraph），从而预测下一次迭代任务的最优并行度。针对批处理中的迭代作业，我们提出了一种基于增强GCN + GRU的算子并行度预测模型，用于预测下一次迭代任务的并行度；针对流处理中的迭代作业，本发明提出了一种感知剩余计算资源的下一次迭代任务最大并行度预测模型。
Smart Images
Figure CN122985407A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of resource scheduling technology, specifically a method for granular partitioning of computing resources in a data lake environment. Background Technology
[0002] With the continuous development of big data, cloud computing, and artificial intelligence technologies, data lakes, as a data management architecture that can store and process any type or quantity of data in a fully faithful manner across local, cloud, and edge computing systems in real time or in batches, have been widely adopted by numerous internet companies. Data lakes store large amounts of unstructured and structured data in their native format, offering better scalability and agility, thus becoming the foundation and key to big data management and analysis.
[0003] Apache Flink, as a distributed computing engine integrating stream and batch processing, has become the core computing support for real-time data lake systems. Unlike the traditional batch processing-dominated model of data lakes that "store first, then compute," real-time data lakes based on Flink can: receive real-time data from multiple sources such as business databases (CDC), message queues (Kafka), and edge logs; and perform value-added processing such as cleaning, deduplication, dimension completion, and format conversion before writing data to the lake tables. Simultaneously, it supports routine iterative governance operations such as compaction merging, small file optimization, incremental data aggregation, and snapshot management within the lake tables, transforming the data lake from a "static storage pool" into a "real-time available lakehouse analytics core," truly enabling querying upon data entry and analysis immediately upon production.
[0004] However, setting optimal parallelism for Flink operators and achieving reasonable granularity of computing resources in the end-to-end data lake operation scenario presents unique challenges far exceeding those of general distributed computing scenarios. Data lake clusters are characterized by the mixed deployment of various heterogeneous jobs, drastic fluctuations in the scale of inbound data, the normalization of dedicated iterative jobs within the lake, and a significant conflict between multi-tenant isolation and elasticity requirements. Furthermore, the storage-compute separation architecture suffers from a lack of storage-compute collaborative optimization, and traditional static configurations cannot adapt to dynamic scenarios. Current mainstream optimization methods are mostly designed for general scenarios and are not deeply adapted to the characteristics of data lake architectures. They lack dedicated optimization mechanisms, and adjusting parallelism requires restarting jobs, which can easily interrupt the inbound data flow and disrupt the consistency of lake surface transactions, failing to meet the production-level operational requirements of data lakes.
[0005] In summary, there is an urgent need for a method for real-time data lake operations that can dynamically and adaptively optimize the parallelism of Flink operators without restarting tasks. This method should automatically sense runtime resource and performance changes, accurately assess the computational requirements of different operators, and achieve optimal resource utilization while ensuring low latency and high stability. This would enable the real-time data lake system to be both engineering-feasible and able to continuously maintain high efficiency, low cost, and strong elasticity in real-world production environments. Summary of the Invention
[0006] To address the shortcomings of existing technologies, the present invention aims to propose a method for granular partitioning of computing resources in a data lake environment, comprising: Step 1: Obtain the execution mode of the Flink job during multiple iterations; Step 2: When the execution mode is batch processing, predict the parallelism of all operators in each round of Flink job to obtain the optimal parallelism of all operators in the next round of Flink job, and use the optimal parallelism as the granularity of the operator. Step 3: When the execution mode is stream processing, the real-time data entering the lake is divided into data segments through multiple windows. Then, the parallelism of each window is predicted to obtain the maximum parallelism of the next round of Flink job window. The maximum parallelism is used as the granularity of the operator.
[0007] Optionally, step 2 includes: Step 2.1: Determine if the current round number k of the Flink job is greater than 10. If not, proceed to step 2.2; if yes, proceed to step 2.3. Step 2.2: Obtain the CPU utilization of all compute nodes within the data lake. Memory utilization Disk I / O utilization Bandwidth utilization Then, a multi-resource threshold heuristic algorithm is used to predict the optimal parallelism of all operators in the (k+1)th round, specifically implemented through the following formula: ; in, These represent the preset target CPU utilization, target memory utilization, target disk I / O utilization, and target bandwidth utilization, respectively. These represent the weights for CPU utilization, memory utilization, disk I / O utilization, and bandwidth utilization, respectively. This represents the optimal parallelism of all operators in the (k+1)th round; Step 2.3: For the k-th round of the Flink job, predict the optimal parallelism of all operators in the (k+1)-th round using graph convolutional networks and gated recurrent units.
[0008] Optionally, step 2.3 includes: Step 2.3.1: For the k-th round of Flink job, construct the job graph; Obtain the operator sequence of the Flink job, treat the operators as vertices, and the data flow between operators as edges to obtain the job graph; where vertices have vertex attributes and edges have edge attributes. Step 2.3.2: Based on the preset multi-tenant quota limit within the data lake, normalize all vertex attributes and stack them to obtain the vertex attribute embedding matrix. After normalizing the edge attributes and stacking them, the adjacency matrix is obtained. ; Step 2.3.3: Embed vertex attributes into the matrix and adjacency matrix enter Layered graph convolutional networks are used to obtain spatial structure features. ; Step 2.3.4: Spatial structural features Input the gated recurrent unit to obtain the joint prediction result. The joint prediction result includes the predicted vertex attributes of all operators in the (k+1)th round of the Flink job; Step 2.3.5: Obtain the preset upper limit and lower limit of operator parallelism, and obtain the predicted parallelism of each operator in the joint prediction result, and then determine the optimal parallelism of all operators in the (k+1)th round, and use the optimal parallelism as the granularity.
[0009] Optionally, the vertex and edge attributes described in step 2.3.1 are obtained in the following way: Obtain the parallelism of each operator in round k; obtain the CPU utilization, memory utilization, disk I / O utilization, and bandwidth utilization of the computing node where the operator is located. The parallelism, CPU utilization, memory utilization, disk I / O utilization, and bandwidth utilization of the operator constitute the vertex attributes, and the data flow in the data lake between operators is used as the edge attributes.
[0010] Optionally, the parameters of the graph convolutional network and the gated recurrent unit are updated according to a loss function, which is calculated based on the joint prediction results and is specifically implemented using the following formula: ; in, Represents the loss function. i Indicates the index of the operator, N This represents the total number of operators. j This represents the index of each dimension in the joint prediction result. d This represents the total number of dimensions in the joint prediction results. Indicates the first i The first prediction result of the joint prediction of the operators j Loss weights in each dimension, Indicates the first i The first prediction result of the joint prediction of the operators j Target utilization rate in each dimension Indicates the firsti The first prediction result of the joint prediction of the operators j The values of each dimension.
[0011] Optionally, the optimal parallelism of all operators in the (k+1)th round described in step 2.3.5 is determined in the following way: When the predicted parallelism is greater than the upper limit of parallelism, the upper limit of parallelism is taken as the optimal parallelism of the operator. When the predicted parallelism is less than the lower limit of parallelism, the lower limit of parallelism is taken as the optimal parallelism of the operator. When the predicted parallelism is greater than or equal to the lower limit of parallelism and less than or equal to the upper limit of parallelism, the predicted parallelism is rounded up to obtain the optimal parallelism, and then the optimal parallelism of all operators in the (k+1)th round is obtained.
[0012] Optionally, step 3 includes: Step 3.1: Define the index of the number of times each window operator sequence processes the data segment as k, and define the index of the number of times all window operator sequences process the data segment as t; Step 3.2: Determine if t is greater than 50. If not, proceed to step 3.3; if yes, proceed to step 3.4. Step 3.3: Using a heuristic algorithm oriented towards remaining resources, calculate the maximum parallelism of the Flink job in the k-th round for each window w, specifically implemented through the following formula: ; in, These represent the preset target remaining CPU resources, target remaining memory resources, target remaining disk I / O resources, and target remaining bandwidth resources, respectively. For the corresponding weights; These represent the remaining CPU resources, remaining memory resources, remaining disk I / O resources, and remaining bandwidth resources of all computing nodes within the data lake, respectively. Step 3.4: Obtain the parallelism, CPU remaining resources, memory remaining resources, disk I / O remaining resources, and bandwidth remaining resources of each operator in the operator sequence at round k to form new vertex attributes; construct an adjacency matrix, and pass the new vertex attributes and adjacency matrix through a graph convolutional network and a gated recurrent unit in sequence to obtain the optimal parallelism of all operators in round k+1. The maximum value of the optimal parallelism of all operators is taken as the maximum parallelism of the window, and the maximum parallelism is taken as the granularity of the operator.
[0013] The beneficial effects of adopting the above technical solution are as follows: By combining the collected cluster resource utilization information, the computational logic of Flink iterative tasks is modeled as a time-varying job graph, thereby predicting the optimal parallelism of the next iterative task. For iterative jobs in batch processing, we propose an operator parallelism prediction model based on enhanced GCN + GRU to predict the parallelism of the next iterative task; for iterative jobs in stream processing, this invention proposes a maximum parallelism prediction model for the next iterative task that is aware of remaining computing resources.
[0014] This invention, based on a heartbeat mechanism and the topology status of data lake operations, collects comprehensive online resource load information and operator execution status information across the data lake's compute and storage layers. The collected metrics not only cover compute layer metrics such as node CPU utilization, memory usage, disk I / O throughput, and network bandwidth utilization, but also include new data lake-specific metrics: lake table write / read throughput, object storage IOPS and access latency, lake table compaction job execution efficiency, partition data scan volume, multi-tenant resource quota utilization, and lake table file operation time. This module ensures that every resource-granularity decision is based on a quantifiable and feedback-enabled real-world operational state of the entire data lake compute-storage stack.
[0015] After obtaining the comprehensive resource data within the aforementioned lake, this invention combines a proposed dynamic granularity segmentation strategy with a prediction model for data lakes to achieve online adaptive partitioning of the granularity of computing resources for multi-source heterogeneous operations in the data lake. The partitioning strategy not only covers operator-level granularity but also dynamically adjusts the time window for data lake entry operations and the batch-level granularity of batch processing operations within the lake. For the iterative operation scenarios unique to data lakes, this invention uses a "lake surface time-dependent subtask graph" as input abstraction, incorporating changes in lake surface metadata and data dependencies between iteration rounds into the model, making the adjustment of computing resource granularity both traceable and predictable across iterations. Through this method, the problems of resource over-provisioning, under-provisioning, and tenant contention in multi-operation mixed deployment scenarios in data lakes can be completely avoided in traditional experience-based resource allocation.
[0016] After adjusting the resource granularity, this invention also maintains the consistency of task state and lake table metadata at the synchronization node of Flink consistency barrier and data lake table transaction commit. This ensures the exactly-Once semantics of the data inflow link and the transaction continuity of operations within the lake before and after the resource granularity change, achieving seamless granularity switching without interrupting business, restarting tasks, losing computation state, or disrupting lake table transactions. During the repartitioning process, this module performs necessary persistence and recovery of incremental inflow data, iteration context, system snapshots, lake table metadata versions, and intermediate task output states to support subsequent scheduling, rollback, and elastic iteration, fully adapting to the production-level requirements of 24 / 7 uninterrupted operation of the data lake. Attached Figure Description
[0017] Figure 1 This is a flowchart illustrating a method for granular partitioning of computing resources in a data lake environment, as described in an embodiment of the present invention. Figure 2 This is a schematic diagram illustrating multiple iterations of a Flink job in an embodiment of the present invention; Figure 3 This is a schematic diagram of the overall structure of the DAPIC framework in an embodiment of the present invention; Figure 4 This is a schematic diagram of the architecture of the batch processing parallelism prediction model in an embodiment of the present invention. Figure 5 This is a schematic diagram of the architecture of the parallelism prediction model in stream processing in an embodiment of the present invention. Detailed Implementation
[0018] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.
[0019] To address the problems of existing technologies, this invention provides a method for granular partitioning of computing resources in a data lake environment, applicable to iterative computing scenarios within a data lake environment. This method proposes a Dynamic Adjustment of Parallelism for Iterative Computing (DAPIC) framework. For real-time data lake entry and in-lake streaming analysis jobs with window periods in the data lake, the DAPIC framework introduces a parallelism adjustment mechanism that is aware of the remaining computing resources in the cluster. For in-lake batch processing iterative jobs (such as lake table compaction merging, in-lake batch graph computation, lake table partition aggregation, etc.), the DAPIC framework introduces a dynamic operator parallelism prediction mechanism based on an enhanced GCN+GRU model. Combined with... Figure 1 This may include the following steps: Step 1: Combining Figure 2 During multiple iterations of a Flink job, the execution mode of the Flink job is obtained; Step 2: When the execution mode is batch processing, predict the parallelism of all operators in each round of the Flink job to obtain the optimal parallelism of all operators in the next round of the Flink job, and use the optimal parallelism as the granularity of the operator; combined with Figure 3 and Figure 4 Specifically, it includes the following steps: Step 2.1: Determine if the current round number k of the Flink job is greater than 10. If not, proceed to step 2.2; if yes, proceed to step 2.3. Step 2.2: Obtain the CPU utilization of all compute nodes within the data lake. Memory utilization Disk I / O utilization Bandwidth utilization .
[0020] Specifically, during the initialization phase of Flink heartbeat detection, the scheduler (such as JobMaster) builds a resource monitoring and management terminal for each compute node (TaskManager). The resource monitoring and management terminal is the heartbeat manager created by the scheduler JobMaster calling the createHeartbeatManagerSender method of the HeartbeatServices class. Its role is to uniformly manage the heartbeat sending and receiving of each compute node (TaskExecutor) connected to the JobMaster, and periodically activate timers, scan the managed compute nodes, activate resource acquisition agents, collect resource load information such as CPU, memory, disk I / O, and bandwidth of each compute node and its TaskSlot execution unit, and establish acquisition sessions for TaskSlot and other execution units.
[0021] The scheduler periodically sends heartbeat requests to compute nodes via remote RPC, and the agent collects metrics such as CPU, memory, disk I / O, and bandwidth in real time, then aggregates these metrics to form a global resource view. For example, in Flink, heartbeat detection is implemented through the HeartbeatServices class to collect resource utilization data.
[0022] In this process, after a new compute node joins the cluster, the agent registers with the master node via RPC. Specifically, after the new compute node (TaskExecutor) is assigned to the JobMaster, it proactively initiates registration with the master node JobMaster via RPC (Remote Procedure Call). The compute node then informs the JobMaster of its hardware configuration and resource usage. Upon receiving the remote RPC call, the JobMaster first processes the local registration and then adds the compute node to its monitoring target set, thus forming the initial resource model.
[0023] The multi-resource threshold heuristic algorithm DoPI is used to predict the optimal parallelism of all operators in the (k+1)th round, specifically implemented by the following formula: ; in, These represent the preset target CPU utilization, target memory utilization, target disk I / O utilization, and target bandwidth utilization, respectively. These represent the weights for CPU utilization, memory utilization, disk I / O utilization, and bandwidth utilization, respectively. This represents the optimal parallelism of all operators in the (k+1)th round; Step 2.3: For the k-th round of the Flink job, predict the optimal parallelism of all operators in the (k+1)-th round using graph convolutional networks and gated recurrent units; Step 2.3.1: For the k-th round of Flink job, construct the job graph; Obtain the operator sequence of the Flink job, which contains multiple operators. Treat the operators as vertices and the data flow between operators as edges to obtain the time-dependent subJobGraph. ,in, For each iteration operator vertex set, Let k be the set of data flow edges within the lake, and k be the iteration round number (Step Function number).
[0024] The vertices have vertex attributes, and the edges have edge attributes. The vertex attributes and edge attributes are obtained in the following ways: Obtain the parallelism of each operator in round k; obtain the CPU utilization, memory utilization, disk I / O utilization, and bandwidth utilization of the computing node where the operator is located. The parallelism, CPU utilization, memory utilization, disk I / O utilization, and bandwidth utilization of the operator constitute the vertex attributes, and the data flow in the data lake between operators is used as the edge attributes. That is to say, each vertex Indicates the first The first step in an iteration step function (i.e., the computational logic unit that is repeatedly executed in each iteration) Each operator carries five-dimensional attributes: c represents the parallelism of the operator. These represent the CPU utilization, memory utilization, disk I / O utilization, and bandwidth utilization of the computing node within the lake where the operator resides; edge attributes. Indicates the first Data in the round from the operator To the operator The data flow within the lake is used to represent the trajectory of operator performance changes across rounds.
[0025] It should be noted that when k=1, the parallelism of the operator is truncated according to the upper / lower limit of the operator parallelism set by the user and rounded up to a positive integer as the initial granularity configuration of the operator. When k is greater than 1, the parallelism of each round is the optimal parallelism predicted in the previous round.
[0026] In this invention, the operator sequence is derived from the JobGraph, which Flink abstracts into a JobGraph upon job submission. This JobGraph is parsed using a Netty-based job acquisition agent, and the operator sequence is extracted according to the data flow dependencies between operators. Furthermore, this invention can extract metadata, including operator type, data format, execution mode (batch / stream processing), operator parallelism, and operator execution time.
[0027] Step 2.3.2: Based on the preset multi-tenant quota limit within the data lake, normalize all vertex attributes and stack them to obtain the vertex attribute embedding matrix. After normalizing the edge attributes and stacking them, the adjacency matrix is obtained. ; Among them, a tenant refers to an independent business party or user group that shares the same computing and storage resources in the data lake cluster. The multi-tenant quota limit in the data lake includes the maximum available CPU, maximum available memory, maximum available disk I / O, and maximum available bandwidth pre-set by the data lake platform for each tenant. Normalization is performed based on this limit, which makes the attribute data of different tenants and different configuration nodes comparable.
[0028] Step 2.3.3: Embed vertex attributes into the matrix and adjacency matrix enter A layered graph convolutional network (GCN; a classic graph neural network structure in this field; this invention directly adopts its standard convolution propagation rules without modifying the network structure itself) is used to obtain spatial structure features. ; In this context, the l-th layer of the graph convolutional network (l = 0, 1, …, L) The convolution process of 1) is as follows: ; in, For the first l Layer trainable weight matrix, For the first l Layer output, initial input ; Let be the sigmoid activation function. The final output of a multi-layer GCN is denoted as . This characterizes the spatial structure features of the lake surface sub-task graph in the current iteration round. The above process is denoted as a function. This is for reference in subsequent steps.
[0029] Step 2.3.4: Spatial structural features Input the gated recurrent unit to obtain the joint prediction result. The joint prediction result includes the predicted vertex attributes of all operators in the (k+1)th round of the Flink job; ; ; ; ; in, To update the door, To reset the door, It is the sigmoid activation function. , , These are the trainable weight matrices for the update gate, the trainable weight matrix for the reset gate, and the trainable weight matrix for the candidate hidden states, respectively. , , These are the bias vectors for the update gate, the reset gate, and the candidate hidden state, respectively. For the first The hidden state of the wheel, For the first The hidden state of the wheel; among which, For joint forecast results ; The parameters of the graph convolutional network and the gated recurrent unit are iteratively updated using gradient descent based on the loss function. The loss function is calculated based on the joint prediction results and is specifically implemented through the following formula: ; in, Represents the loss function. i Indicates the index of the operator. N This represents the total number of operators. j This represents the index of each dimension in the joint prediction result. d This represents the total number of dimensions in the joint prediction results. Indicates the first i The first prediction result of the joint prediction of the operators j Loss weights in each dimension, Indicates the first i The first prediction result of the joint prediction of the operators j Target utilization rate in each dimension Indicates the first i The first prediction result of the joint prediction of the operators j Values for each dimension; It should be noted that the first dimension of the joint prediction result is the prediction parallelism. The loss function does not constrain this dimension, so the dimension index j starts from 2 to d. The dimensions from 2 to d are the predicted CPU utilization, predicted memory utilization, predicted disk I / O utilization, and predicted bandwidth utilization, respectively.
[0030] The parameters of the graph convolutional network and the gated recurrent unit include the weights and biases mentioned earlier, for example... , , , and the corresponding bias vector.
[0031] By minimizing Until The algorithm converges or drops below a set threshold and is subject to user-defined upper / lower limits on operator parallelism. It then solves for the optimal parallelism of the next round of operators under three constraints: low latency, low resource cost, and tenant quota.
[0032] Step 2.3.5: Obtain the preset upper limit and lower limit of operator parallelism, and obtain the prediction parallelism of each operator in the joint prediction result, and then determine the optimal parallelism of all operators in the (k+1)th round, and use the optimal parallelism as the granularity. When the predicted parallelism is greater than the upper limit of parallelism, the upper limit of parallelism is taken as the optimal parallelism of the operator. When the predicted parallelism is less than the lower limit of parallelism, the lower limit of parallelism is taken as the optimal parallelism of the operator. When the predicted parallelism is greater than or equal to the lower limit of parallelism and less than or equal to the upper limit of parallelism, the predicted parallelism is rounded up to obtain the optimal parallelism, and then the optimal parallelism of all operators in the (k+1)th round is obtained. Furthermore, this invention safely injects the optimization results into the scheduler (such as Flink's JobMaster) at the barrier synchronization point, achieving hot updates and ensuring state consistency. The barrier synchronization point refers to the consistency barrier moment between each iteration in the Flink iterative computation model: at this point, all operator tasks of the previous iteration have been completed, and the tasks of the next iteration have not yet started. This allows for obtaining complete statistical information from the previous iteration without needing to process intermediate task states, making it a safe time to adjust parallelism. The "scheduler" refers to Flink's JobMaster, which is responsible for allocating job operators to each TaskSlot according to parallelism and driving their execution. "Hot update" refers to directly injecting the optimal parallelism configuration into the scheduler at the barrier synchronization point without restarting the job or interrupting the lake entry link, and making it effective in the next iteration, thus ensuring consistency between states and achieving seamless granularity switching. The current round's state trajectory (such as resource consumption and graph snapshots) is saved at the barrier synchronization point, and the granularity of the next round is predicted based on historical data, applied in a hot update manner. Step 3: Combining Figure 5 When the execution mode is stream processing, the real-time data entering the lake is divided into data segments through multiple windows. Then, the parallelism of each window is predicted to obtain the maximum parallelism of the next round of Flink job window. The maximum parallelism is used as the granularity of the operator. In stream processing, data arrives continuously in windows. A window refers to a segment of data obtained by dividing a continuous data stream according to time or data volume. The data in each window w is processed using the same logic, which consists of several operators and is abstracted as a lake surface time-correlation sub-task graph. (The operator runs on each computing node within the lake, and k is still the iteration round number within the window). Since the cluster needs to process data from multiple windows simultaneously, the Step Functions in different windows will be executed alternately. All Step Functions in all windows are uniformly numbered according to their actual execution order in the cluster, thus obtaining the global execution sequence number t of the cross-window Step Function (therefore t is the sequence number in the global execution sequence: the same Step Function execution corresponds to both the kth iteration within its window w and the tth execution in the global sequence; t and k represent the correspondence between the global sequence number and the local round number within the window).
[0033] Step 3.1: Define the index of the number of times each window operator sequence processes the data segment as k, and define the index of the number of times all window operator sequences process the data segment as t; Step 3.2: Determine if t is greater than 50. If not, proceed to step 3.3; if yes, proceed to step 3.4. Step 3.3: Using a heuristic algorithm oriented towards remaining resources, calculate the maximum parallelism of the Flink job in the k-th round for each window w, specifically implemented through the following formula: ; in, These represent the preset target remaining CPU resources, target remaining memory resources, target remaining disk I / O resources, and target remaining bandwidth resources, respectively. For the corresponding weights; These represent the remaining CPU resources, remaining memory resources, remaining disk I / O resources, and remaining bandwidth resources of all computing nodes within the data lake, respectively. Step 3.4: Obtain the parallelism, CPU remaining resources, memory remaining resources, disk I / O remaining resources, and bandwidth remaining resources of each operator in the operator sequence at round k to form new vertex attributes; construct the adjacency matrix (in the same way as the previous adjacency matrix construction), and pass the new vertex attributes and adjacency matrix through the graph convolutional network and gated recurrent unit in sequence to obtain the optimal parallelism of all operators in round k+1. The maximum value of the optimal parallelism of all operators is taken as the maximum parallelism of the window, and the maximum parallelism is taken as the granularity of the operator; In stream processing, the parallelism is not adjusted per operator, but only the maximum parallelism is adjusted by window. This ensures low latency in the inbound link while controlling TaskSlot occupancy and avoiding disruption of the processing continuity of other inbound windows.
[0034] This invention also includes multi-condition triggered adaptive closed-loop control and feedback optimization. Based on events (such as iteration completion) or environmental changes (such as resource fluctuations), an adjustment process is triggered to form a "perception-decision-execution-feedback" closed loop, improving resource utilization efficiency. Here, "event" specifically refers to execution nodes such as the arrival of a barrier synchronization point or the completion of a single iteration; "environmental change" specifically refers to changes in the operational state, such as resource utilization indicators (CPU, memory, disk I / O, bandwidth, etc.) collected in step 2 exceeding preset thresholds, or adjustments to multi-tenant resource quotas; that is, changes in the aforementioned collected data; "adjustment process" refers to the process triggered by this, re-executing step 2 or step 3 to generate and hot-update the next round of granular configuration, thus forming a "perception-decision-execution-feedback" closed loop.
[0035] Through the above steps, this invention achieves dynamic and precise partitioning of computing resources in a data lake environment, significantly improving task efficiency and resource utilization.
[0036] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.
Claims
1. A method for granular partitioning of computing resources in a data lake environment, characterized in that, include: Step 1: Obtain the execution mode of the Flink job during multiple iterations; Step 2: When the execution mode is batch processing, predict the parallelism of all operators in each round of Flink job to obtain the optimal parallelism of all operators in the next round of Flink job, and use the optimal parallelism as the granularity of the operator. Step 3: When the execution mode is stream processing, the real-time data entering the lake is divided into data segments through multiple windows. Then, the parallelism of each window is predicted to obtain the maximum parallelism of the next round of Flink job window. The maximum parallelism is used as the granularity of the operator.
2. The method for granular partitioning computing resources in a data lake environment according to claim 1, characterized in that, Step 2 includes: Step 2.1: Determine if the current round number k of the Flink job is greater than 10. If not, proceed to step 2.2; if yes, proceed to step 2.
3. Step 2.2: Obtain the CPU utilization of all compute nodes within the data lake. Memory utilization Disk I / O utilization Bandwidth utilization Then, a multi-resource threshold heuristic algorithm is used to predict the optimal parallelism of all operators in the (k+1)th round, specifically implemented through the following formula: ； in, These represent the preset target CPU utilization, target memory utilization, target disk I / O utilization, and target bandwidth utilization, respectively. These represent the weights for CPU utilization, memory utilization, disk I / O utilization, and bandwidth utilization, respectively. This represents the optimal parallelism of all operators in the (k+1)th round; Step 2.3: For the k-th round of the Flink job, predict the optimal parallelism of all operators in the (k+1)-th round using graph convolutional networks and gated recurrent units.
3. The method for granular partitioning computing resources in a data lake environment according to claim 2, characterized in that, Step 2.3 includes: Step 2.3.1: For the k-th round of Flink job, construct the job graph; Obtain the operator sequence of the Flink job, treat the operators as vertices, and the data flow between operators as edges to obtain the job graph; where vertices have vertex attributes and edges have edge attributes. Step 2.3.2: Based on the preset multi-tenant quota limit within the data lake, normalize all vertex attributes and stack them to obtain the vertex attribute embedding matrix. After normalizing the edge attributes and stacking them, the adjacency matrix is obtained. ; Step 2.3.3: Embed vertex attributes into the matrix and adjacency matrix enter Layered graph convolutional networks are used to obtain spatial structure features. ; Step 2.3.4: Spatial structural features Input the gated recurrent unit to obtain the joint prediction result. The joint prediction result includes the predicted vertex attributes of all operators in the (k+1)th round of the Flink job; Step 2.3.5: Obtain the preset upper limit and lower limit of operator parallelism, and obtain the predicted parallelism of each operator in the joint prediction result, and then determine the optimal parallelism of all operators in the (k+1)th round, and use the optimal parallelism as the granularity.
4. The method for granular partitioning computing resources in a data lake environment according to claim 3, characterized in that, The vertex and edge attributes mentioned in step 2.3.1 are obtained in the following way: Obtain the parallelism of each operator in round k; obtain the CPU utilization, memory utilization, disk I / O utilization, and bandwidth utilization of the computing node where the operator is located. The parallelism, CPU utilization, memory utilization, disk I / O utilization, and bandwidth utilization of the operator constitute the vertex attributes, and the data flow in the data lake between operators is used as the edge attributes.
5. The method for granular partitioning computing resources in a data lake environment according to claim 3, characterized in that, The parameters of the graph convolutional network and the gated recurrent units are updated according to a loss function, which is calculated based on the joint prediction results and is specifically implemented using the following formula: ； in, Represents the loss function. i Indicates the index of the operator, N This represents the total number of operators. j This represents the index of each dimension in the joint prediction result. d This represents the total number of dimensions in the joint prediction results. Indicates the first i The first prediction result of the joint prediction of the operators j Loss weights in each dimension, Indicates the first i The first prediction result of the joint prediction of the operators j Target utilization rate in each dimension Indicates the first i The first prediction result of the joint prediction of the operators j The values of each dimension.
6. The method for granular partitioning computing resources in a data lake environment according to claim 3, characterized in that, The optimal parallelism of all operators in the (k+1)th round mentioned in step 2.3.5 is determined in the following way: When the predicted parallelism is greater than the upper limit of parallelism, the upper limit of parallelism is taken as the optimal parallelism of the operator. When the predicted parallelism is less than the lower limit of parallelism, the lower limit of parallelism is taken as the optimal parallelism of the operator. When the predicted parallelism is greater than or equal to the lower limit of parallelism and less than or equal to the upper limit of parallelism, the predicted parallelism is rounded up to obtain the optimal parallelism, and then the optimal parallelism of all operators in the (k+1)th round is obtained.
7. The method for granular partitioning computing resources in a data lake environment according to claim 1, characterized in that, Step 3 includes: Step 3.1: Define the index of the number of times each window operator sequence processes the data segment as k, and define the index of the number of times all window operator sequences process the data segment as t; Step 3.2: Determine if t is greater than 50. If not, proceed to step 3.3; if yes, proceed to step 3.
4. Step 3.3: Using a heuristic algorithm oriented towards remaining resources, calculate the maximum parallelism of the Flink job in the k-th round for each window w, specifically implemented through the following formula: ； in, These represent the preset target remaining CPU resources, target remaining memory resources, target remaining disk I / O resources, and target remaining bandwidth resources, respectively. For the corresponding weights; These represent the remaining CPU resources, remaining memory resources, remaining disk I / O resources, and remaining bandwidth resources of all computing nodes within the data lake, respectively. Step 3.4: Obtain the parallelism, CPU remaining resources, memory remaining resources, disk I / O remaining resources, and bandwidth remaining resources of each operator in the operator sequence at round k to form new vertex attributes; construct an adjacency matrix, and pass the new vertex attributes and adjacency matrix through a graph convolutional network and a gated recurrent unit in sequence to obtain the optimal parallelism of all operators in round k+1. The maximum value of the optimal parallelism of all operators is taken as the maximum parallelism of the window, and the maximum parallelism is taken as the granularity of the operator.