Flink job elastic scaling method and device, electronic equipment and storage medium

Abstract

The embodiment of the application provides a kind of Flink job elasticity scaling method, device, electronic equipment and storage medium, the method comprises the following steps in current detection period: based on at least one root node, any root node performance value, at least one root node, any root node production rate is calculated first performance utilization rate;Determine at least one second parallelism according to at least one first performance utilization rate, first parallelism and target performance utilization rate, determine third parallelism based on at least one second parallelism, according to the relationship between first parallelism and third parallelism, capacity adjustment is carried out to Flink job. Through the embodiment shown in the application, only according to performance utilization rate, the recommended parallelism of overall Flink job is determined, that is, the elasticity scaling of Flink job can be carried out, and the stability of Flink system job is improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and specifically to a method, apparatus, electronic device, and storage medium for flexible scaling of Flink jobs. Background Technology

[0002] Apache Flink is an open-source platform for processing streaming and batch data, providing users with high-throughput, low-latency data processing services. A Flink application typically consists of multiple Vertices, each containing at least one Operator. Each Operator can have multiple Task instances executing in parallel. The Task instances collectively complete the Operator's logic. Vertices are connected by edges, forming a data flow graph topology, and the data flow paths are defined based on this topology.

[0003] While a static scaling method for Flink systems has been proposed in related technologies, it requires adjusting the system using multiple parameters during the static scaling process. Furthermore, conflicts can easily arise during this adjustment process, resulting in suboptimal job scaling adjustment in Flink systems. Summary of the Invention

[0004] In view of this, this application provides a simulation scene generation method, apparatus and equipment to help solve the technical problem of poor Flink job scaling adjustment in the prior art and improve the stability of Flink system elastic scaling adjustment.

[0005] Firstly, the Flink job elastic scaling method provided in this application embodiment performs the following steps within the current detection cycle:

[0006] The first performance utilization rate is calculated based on the performance value of any node in at least one root node and the production rate of any node in at least one root node; the first performance utilization rate is: the performance utilization rate of any vertex in at least one root node under the current Flink job;

[0007] At least one second parallelism is determined based on at least one first performance utilization, a first parallelism, and a target performance utilization. The first parallelism is the current parallelism of the Flink job, and the second parallelism is the recommended parallelism of any vertex of at least one root node in the current Flink job.

[0008] The third parallelism is determined based on at least one second parallelism, and the third parallelism is: the maximum parallelism that exists in at least one second parallelism.

[0009] The capacity of Flink jobs is adjusted based on the relationship between the first parallelism and the third parallelism.

[0010] The embodiments shown in this application demonstrate that the recommended parallelism of the overall Flink job can be determined solely based on performance utilization, without relying on other job metrics, thereby enabling elastic scaling of Flink jobs and improving the stability of Flink system jobs.

[0011] One possible approach is to adjust the capacity of a Flink job based on the relationship between the first and third degrees of parallelism, including the following steps:

[0012] If the first parallelism is greater than the third parallelism, the Flink job will be scaled down.

[0013] If the first parallelism is less than the third parallelism, the Flink job will be scaled up.

[0014] If the first parallelism is equal to the third parallelism, then the Flink job is executed according to the current resource slot capacity.

[0015] One possible approach is to shrink or expand a Flink job, which involves the following steps:

[0016] The resource slot capacity is determined based on the ratio of the first parallelism to the third parallelism.

[0017] One possible approach is to further include, before calculating the first performance utilization rate based on the performance value of any node in at least one root node and the production rate of any node in at least one root node:

[0018] Calculate the performance value of any one of the at least one root nodes.

[0019] One possible approach is to compute the performance value of any one of the at least one root nodes by including the following steps:

[0020] Calculate the performance value of at least one target vertex;

[0021] Based on the performance value of at least one target vertex, the consumption rate of at least one target vertex, and the consumption rate of any one of the at least one root nodes, calculate the performance value of at least one root node. The target vertex is the downstream node of any one of the at least one root nodes.

[0022] Select at least one root node from at least one root node performance value, wherein the performance value of any one root node is the minimum value among the performance values ​​of at least one target vertex.

[0023] One possible approach is to calculate the performance value of at least one target vertex by including:

[0024] Select at least one first target execution unit and calculate the performance value of at least one first target execution unit, wherein the first target execution unit is the execution unit of any target vertex among at least one target vertex;

[0025] A second target execution unit is selected from at least one first target execution unit, wherein the second target execution unit is the execution unit that reaches the bottleneck first among at least one first target execution unit;

[0026] The performance value of any target vertex among at least one target vertex is determined based on the performance value of the second target execution unit, the consumption rate of the second target execution unit, and the consumption rate of any target vertex among at least one target vertex. The performance value of the second target execution unit exists within the performance value of at least one first target execution unit.

[0027] One possible approach is to calculate the first performance utilization rate based on the performance value of any node in at least one root node and the production rate of any node in at least one root node.

[0028] The performance utilization of any node in at least one root node under the current Flink job is calculated using the following formula:

[0029]

[0030] utilization source —First performance utilization;

[0031] production rate source —The production rate of the root node;

[0032] throughout source —Root node performance value.

[0033] One possible approach is to calculate the second parallelism using the following formula in the step of determining the second parallelism based on the first performance utilization, the first parallelism, and the target performance utilization:

[0034]

[0035] parallelism recommend —Second degree of parallelism;

[0036] utilization source —First performance utilization;

[0037] utilization target—Target performance utilization;

[0038] parallelism current —First degree of parallelism.

[0039] One possible approach is to calculate the production rate of any one of the at least one root node as follows:

[0040] Determine the consumption rate of any node in at least one root node, and the backlog growth rate of any node in at least one root node.

[0041] In at least one root node, the production rate of any one root node is the sum of the consumption rate of any one root node and the backlog growth rate of any one root node.

[0042] One possible approach is to perform the following steps in the next detection cycle:

[0043] The second performance utilization rate is calculated based on the performance value of any node in at least one root node and the production rate of any node in at least one root node; the second performance utilization rate is: the performance utilization rate of any vertex in at least one root node in the next cycle of the current detection cycle.

[0044] Determine whether the difference between the second performance utilization rate and the first performance utilization rate exceeds a threshold.

[0045] If so, the first performance utilization rate is updated based on the second performance utilization rate, and the step of determining at least one second parallelism rate based on the at least one first performance utilization rate, the first parallelism rate, and the target performance utilization rate is performed.

[0046] If not, perform the step of determining at least one second parallelism based on the at least one first performance utilization, the first parallelism, and the target performance utilization.

[0047] Secondly, an embodiment of this application provides a Flink job elastic scaling device, comprising:

[0048] First calculation module: used to calculate the first performance utilization rate based on the performance value of any node in at least one root node and the production rate of any node in at least one root node; the first performance utilization rate is: the performance utilization rate of any vertex in at least one root node under the current Flink job;

[0049] The second computation module is used to determine at least one second parallelism based on at least one first performance utilization, a first parallelism, and a target performance utilization. The first parallelism is the current parallelism of the Flink job, and the second parallelism is the recommended parallelism of any vertex of at least one root node in the current Flink job.

[0050] Determining module: used to determine a third degree of parallelism based on at least one second degree of parallelism, wherein the third degree of parallelism is the maximum degree of parallelism that exists in at least one second degree of parallelism;

[0051] Adjustment module: Used to adjust the capacity of Flink jobs based on the relationship between the first parallelism and the third parallelism.

[0052] Thirdly, the present invention provides an electronic device, comprising:

[0053] At least one processor; and

[0054] At least one memory communicatively connected to the processor, wherein:

[0055] The memory stores program instructions that can be executed by the processor, and the processor can execute the method of the first aspect by calling the program instructions.

[0056] Fourthly, the present invention provides a computer-readable storage medium, a non-transitory computer-readable storage medium storing computer instructions that cause a computer to perform the method as described in the first aspect. Attached Figure Description

[0057] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 A Flink system data flow diagram illustrated in an exemplary embodiment of this application;

[0059] Figure 2 A flowchart of a method for flexible scaling of FLink jobs provided in related technologies.

[0060] Figure 3 Electronic device structure diagram provided in the embodiments of this application

[0061] Figure 4(a) is an exemplary illustration of the static configuration of the Flink system as a function of load, provided in an embodiment of this application.

[0062] Figure 4(b) is an exemplary illustration of the Flink system configuration under elastic scaling as a function of load, provided in an embodiment of this application.

[0063] Figure 5 A busy time distribution per second for a parallel task is shown in an exemplary embodiment provided for this application.

[0064] Figure 6 The elastic scaling system structure of the Flink system provided in the embodiments of this application;

[0065] Figure 7 A detailed flowchart of Flink elastic jobs provided in the embodiments of this application;

[0066] Figure 8 A flowchart of a Flink job elastic scaling method provided in this application embodiment;

[0067] Figure 9 A structural diagram of a Flink job elastic scaling device provided in an embodiment of this application;

[0068] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0069] To better understand the technical solution of this application, the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0070] It should be understood that the described embodiments are merely some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0071] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0072] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0073] First, the terms and concepts used in this application will be explained:

[0074] Vertex: The internal scheduling and execution vertex of a Flink job, representing a node in the data flow graph. It is responsible for sending the input data at the current vertex to downstream nodes. Each vertex includes at least one operator, which can be executed in different execution units. It should also be noted that multiple vertices can be connected by edges to form the topology of the entire data flow graph.

[0075] Operator: A logical unit in a Flink job, representing a specific logical operation for data stream processing, including but not limited to: data transformation, data filtering, and aggregation. Each operator can contain one or more execution units, which can execute tasks in parallel.

[0076] It is worth noting that the operator itself does not perform any task; the task is actually performed by the execution unit described later.

[0077] Task: In Flink, a task is a specific unit of execution. Each unit of execution represents a specific parallel task, and each unit of execution is usually mapped to a resource slot during runtime.

[0078] Here, each execution unit is responsible for executing the tasks of one or more operators, and the execution units process a portion of the input data in parallel and generate corresponding output data.

[0079] Source: The root node of the Flink job flow graph, responsible for connecting external data to the Flink system.

[0080] Production Rate: The speed at which data producers generate messages or records.

[0081] Consumption Rate: The rate at which data consumers take away data.

[0082] Throughout: Performance value, characterizing the speed at which a Flink job can process the number of records or the amount of data. Typically, the performance value is expressed in records per second or MBs per second.

[0083] Utilization: The ratio of production rate to performance value, indicating the degree to which performance is available. Specifically, it can be interpreted as: when the production rate is less than the performance value, at this time, production rate = consumption rate < performance value, indicating that the current performance is not fully utilized.

[0084] When the production rate is greater than the performance value (production rate > consumption rate = performance value), it indicates that the current performance has been fully utilized and can no longer meet the producer's needs.

[0085] Parallelism: In conjunction with the above, Flink consists of multiple tasks (i.e., execution units), meaning each task can be decomposed into multiple parallel instances for execution. Each parallel instance processes a subset of the task's input data. The number of parallel instances per task is the degree of parallelism.

[0086] Slot: A resource slot, the unit of resource allocation in a Flink job. Each slot represents a container of available computational resources. There are usually multiple slots on a task manager, and each slot can execute one or more tasks.

[0087] SlotUtilization: Slot utilization rate, representing the resource utilization rate of a slot.

[0088] Based on the aforementioned background information, the following is a brief overview of the Flink job workflow:

[0089] Apache Flink is an open-source platform for processing streaming and batch data, providing users with high-throughput, low-latency data processing services. A Flink application typically consists of multiple Vertexes, each including at least one Operator. Each Operator can have multiple Task instances executing in parallel. The Task instances collectively complete the Operator's logic. The Vertexes are connected by edges, thus forming a data flow graph topology, and the data flow path is defined based on the topology.

[0090] Please refer to Figure 1 ,by Figure 1 Taking the data flow graph in Flink as an example, Flink receives data from upstream Kafla and HDFS data sources. At this time, the Flink system obtains streaming data and file data, performs calculations in the Flink system through the Operator, and then outputs the calculation results to the downstream system.

[0091] It should be noted that for each operator in the Flink system, the computation process can be decomposed into multiple instances, and each instance can compute in parallel. Each instance processes a portion of the data to improve the throughput of the Flink system. In conjunction with the above, the number of parallel instances corresponds to the aforementioned parallelism.

[0092] In summary, for open-source Flink, the configuration method mainly relies on manually configuring the parallelism of jobs. This configuration method has the following two drawbacks:

[0093] 1) Inability to adjust traffic: First, the Flink system uses static resource configuration, which cannot be modified during operation. However, the traffic of the Flink system during the job is not constant. In other words, the Flink system does not run at a constant traffic during the job. If the Flink system resource configuration is too low, it will cause data backlog when the Flink system traffic is high. If the Flink system configuration is too high, it will lead to resource waste when the traffic is low.

[0094] 2) Debugging requires significant manpower: The precise resource configuration of the Flink system is related to many factors such as environment, traffic volume, data distribution, and job logic complexity. Debugging requires repeated debugging, resulting in a long debugging cycle and high manpower costs.

[0095] In summary, because Flink uses a static configuration method, it cannot adjust for traffic during operation, resulting in unreasonable resource allocation. Furthermore, debugging is time-consuming and requires significant manpower.

[0096] To enable elastic job creation using Flink system resources, please search for... Figure 2 , Figure 2 Here is a flowchart of a Flink job elastic scaling method based on related technologies, which includes the following steps:

[0097] S101: Determine if the work delay index is normal.

[0098] It should be noted that the delay indicator here corresponds to the delay time, which in related technologies means determining whether the delay time exceeds 60 seconds.

[0099] S101a: If the delay indicator is abnormal, determine whether the delay indicator has decreased;

[0100] Specifically, determining whether the latency index has decreased mainly involves determining whether the latency time has shortened. For example, assuming the latency time is 80 seconds, we need to determine whether the latency time has shortened. Specifically, we monitor the latency time in real time. After real-time monitoring, we obtain the real-time latency time and analyze it. If the real-time latency time shows a downward trend, the latency index is considered to have decreased; otherwise, the latency index is considered to have increased.

[0101] Furthermore, if the latency metric decreases, the current Flink job will continue at the current concurrency level.

[0102] If the latency index continues to rise within the preset time period, the concurrency will be increased to twice the original concurrency.

[0103] Furthermore, increasing the concurrency can increase the performance of Flink system jobs. For example, by increasing the concurrency, the performance of the expanded Flink system is doubled.

[0104] S101b: If the latency index is normal, determine whether the proportion of time that the Vertex spends processing data within a preset time period exceeds a first preset value. It should be noted that the first preset value specifically refers to the maximum proportion of time that the Vertex spends processing data for any given Vertex.

[0105] If so, then increase the concurrency.

[0106] After increasing concurrency, slot utilization decreased to 50%.

[0107] If not, further determine whether the CPU (Central Processing Unit) utilization of all threads in the Flink system exceeds the second preset value within the preset time period. It should be noted that the second preset value here refers to the maximum average utilization allowed for any thread.

[0108] If so, then increase the concurrency.

[0109] After reducing concurrency, the average CPU utilization of all threads decreased to 50%.

[0110] If not, determine that the CPU (Central Processing Unit) usage rate of all threads in the Flink system is lower than the third preset value within the preset time period; if so, reduce the concurrency.

[0111] It should be noted that the third preset value here is a generic pointer to the minimum allowed CPU continuity rate for any thread. When the minimum allowed CPU continuity rate for a certain thread is lower than the third preset value, it means that the Flink job configuration is too high for that thread, and there is redundancy in the system (not shown in the figure).

[0112] After reducing concurrency, the actual processing time of CPU and VERTEX increased to 50%.

[0113] It should be noted that the maximum limit for increasing concurrency is the maximum resource set, with a default value of 64 CUs.

[0114] In summary, while a method for elastic scaling of Flink jobs has been proposed in related technologies, using this method for adjusting Flink job capacity requires considering multiple metrics and judging each metric separately, increasing system complexity. Furthermore, conflicts may arise when adjusting different metrics, leading to system instability.

[0115] Secondly, if only the scaling up of indicators is considered, such as scaling down or up based solely on slot utilization and CPU utilization, the scaling up or down space is limited, usually requiring multiple adjustments. Furthermore, when the backlog grows, the slot utilization and CPU utilization cannot reflect the true resource requirements of Flink jobs.

[0116] Finally, when there is a backlog, directly increasing the concurrency to twice the original level results in poor precision in resource adjustment and a rather crude adjustment method.

[0117] To address the aforementioned issues and improve the stability of Flink job scaling and shrinking, and to implement refined management of Flink jobs, this application proposes a Flink job elastic scaling method, apparatus, electronic device, and storage medium.

[0118] Figure 3 A schematic diagram of the structure of the electronic device 100 is shown.

[0119] Electronic device 100 may include processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0120] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0121] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0122] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0123] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0124] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0125] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C buses. The processor 110 can couple to the touch sensor 180K, charger, flash, camera 193, etc., through different I2C bus interfaces. For example, the processor 110 can couple to the touch sensor 180K through the I2C interface, enabling the processor 110 and the touch sensor 180K to communicate through the I2C bus interface, thereby realizing the touch function of the electronic device 100.

[0126] Internal memory 121 can be used to store executable program code, including instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional applications and data processing of electronic device 100 by running instructions stored in internal memory 121 and / or instructions stored in memory located within the processor.

[0127] The following describes a method for elastic scaling of Flink jobs provided in this application:

[0128] based on Figure 1 The data shown is a Flink data flow diagram. Kalfa and HDFS data sources are two sources in the Flink system. Here, for each source, the electronic device calculates its performance utilization based on its performance value and production rate. That is, for the Kalfa root node, the performance utilization of the Kalfa root node is calculated, and for the HDFS data source root node, the performance utilization of the HDFS data source root node is calculated.

[0129] At this point, for electronic devices, the first performance utilization rate is calculated based on the performance value of any node in at least one root node and the production rate of any node in at least one root node; the first performance utilization rate is: the performance utilization rate of any vertex in at least one root node under the current Flink job;

[0130] It should be noted that, in combination Figure 1The Flink job flow diagram shown corresponds to the performance utilization of the HDFS data source root node and the performance utilization of the Kalfa root node.

[0131] In summary, among at least one root node, the performance value of any one root node is... Figure 1 The performance values ​​of the Kalfa root node and the HDFS data source root node in the Flink data flow graph;

[0132] In at least one root node, the production rate of any node is Figure 1 The production rate of the Kalfa root node and the production rate of the HDFS data source root node are shown in the Flink data flow graph.

[0133] It should be noted that, in the embodiments of this application, the API interface of the Flink system can monitor the Flink cluster metrics in real time, and use the Flink cluster metrics to calculate and determine the production rate of the Kalfa root node, the production rate of the HDFS data source root node, the performance value of the Kalfa root node, and the performance value of the HDFS data source root node.

[0134] In other words, the methods for determining the production rate of the Kalfa root node, the production rate of the HDFS data source root node, the performance value of the Kalfa root node, and the performance value of the HDFS data source root node mentioned later in this application are merely one possible implementation method provided by this application and are not intended to limit this application.

[0135] Furthermore, in this step, the performance utilization of any node in at least one root node under the current Flink job is calculated using the following formula:

[0136]

[0137] utilization source —First performance utilization;

[0138] production rate source —The production rate of the root node;

[0139] throughout source —Node performance value.

[0140] In other words, the performance utilization of the Kalfa root node can be determined by dividing the production rate of the Kalfa root node by its performance value. The performance utilization of the HDFS data source root node can be obtained in the same way.

[0141] After obtaining the performance utilization rates of the Kalfa root node and the HDFS data source root node, the recommended parallelism of the Kalfa root node and the HDFS data source root node are calculated respectively. Taking the Kalfa root node as an example, the recommended parallelism of the Kalfa root node can be determined based on the correspondence between the performance utilization rate of the Kalfa root node, the target performance utilization rate of the Kalfa root node, and the current parallelism.

[0142]

[0143] parallelism recommend —Second degree of parallelism;

[0144] utilization source —First performance utilization;

[0145] utilization target —Target performance utilization;

[0146] parallelism current —First degree of parallelism.

[0147] It should be noted that the first parallelism here corresponds to the current parallelism of the Flink system, the second parallelism corresponds to the recommended parallelism of the Kalfa root node, and the first performance utilization corresponds to the performance utilization of the aforementioned Kalfa root node.

[0148] The recommended parallelism of the Kalfa root node can be obtained in this way.

[0149] Similarly, the recommended parallelism of the HDFS data source root node can be obtained.

[0150] In other words, in the embodiments provided in this application, the electronic device determines at least one second degree of parallelism based on at least one first performance utilization rate, a first degree of parallelism, and a target performance utilization rate.

[0151] It should be noted that the first parallelism is the current parallelism of the Flink job, and the second parallelism is the recommended parallelism of at least one root node and any vertex in the current Flink job.

[0152] Thus, the recommended parallelism of the Kalfa root node and the recommended parallelism of the HDFS data source root node are obtained.

[0153] At this point, the recommended parallelism of the Kalfa root node and the recommended parallelism of the HDFS data source root node are compared, and the maximum parallelism between the two is selected as the recommended parallelism of the entire Flink system.

[0154] Alternatively, it can be said that the electronic device determines the third degree of parallelism based on at least one second degree of parallelism.

[0155] It should be noted that the third degree of parallelism is the maximum degree of parallelism that exists in at least one second degree of parallelism.

[0156] For example, if the recommended parallelism of the Kalfa root node is greater than that of the HDFS data source root node, then the recommended parallelism of the Kalfa root node is selected as the third parallelism. Conversely, if the recommended parallelism of the Kalfa root node is less than that of the HDFS data source root node, then the recommended parallelism of the HDFS data source root node is selected as the third parallelism.

[0157] This means obtaining the overall parallelism of the Flink job, and then adjusting the capacity of the Flink job based on the relationship between the first parallelism and the third parallelism.

[0158] By following the steps above, elastic job scaling can be performed on any Flink system based on performance utilization, eliminating the need to adjust Flink job capacity using multiple performance metrics, thus increasing system stability during capacity adjustment.

[0159] In some embodiments, in order to improve the stability of Flink system scaling, the performance utilization rate of the current Flink system root node is updated only if the difference between the performance utilization rate of each root node of Flink obtained at the next collection time and the performance utilization rate of the current Flink system root node exceeds a preset value.

[0160] Specifically, the performance utilization rate calculated in the next detection cycle is compared with the performance utilization rate calculated in this cycle, and the difference is obtained. If the difference is greater than the preset value, the overall parallelism of the Flink job is recalculated using the performance utilization rate calculated in the next detection cycle in the aforementioned manner. If the difference exceeds the preset value, the overall parallelism of the Flink job is recalculated using the performance value calculated in this cycle.

[0161] In some embodiments, the following steps are performed in the next cycle of the current detection cycle:

[0162] The second performance utilization rate is calculated based on the performance value of any node in at least one root node and the production rate of any node in at least one root node; the second performance utilization rate is: the performance utilization rate of any vertex in at least one root node in the next cycle of the current detection cycle.

[0163] Determine whether the difference between the second performance utilization rate and the first performance utilization rate exceeds a threshold.

[0164] If so, the first performance utilization rate is updated based on the second performance utilization rate, and the step of determining at least one second parallelism rate based on the at least one first performance utilization rate, the first parallelism rate, and the target performance utilization rate is performed.

[0165] If not, perform the step of determining at least one second parallelism based on the at least one first performance utilization, the first parallelism, and the target performance utilization.

[0166] Meanwhile, this application implements fine scaling up and down of multiple Flink jobs. Assuming that the overall recommended parallelism of Flink jobs is 3 and the current parallelism of Flink jobs is 6, it indicates that there is redundancy in the Flink system. Therefore, the Kalfa root node is scaled down based on the ratio of the current parallelism to the overall parallelism of the Flink system (the aforementioned third parallelism).

[0167] For example, if the current resource slot utilization rate is 80%, and considering the ratio of the current parallelism to the overall recommended parallelism of the Flink job (2) mentioned above, the resource slot utilization rate should be adjusted to 40% to reduce the capacity.

[0168] For example, if the current parallelism of the Flink system is 3 and the recommended parallelism of the Flink job as a whole is 6, then it means that the resource slot utilization needs to be increased to expand the capacity. The resource slot utilization is 40%, and after expansion, the resource slot utilization will be 80%.

[0169] Of course, in some instances, Flink jobs cannot be scaled up according to the ratio of the current parallelism to the overall recommended parallelism of Flink jobs. For example, if the current parallelism of the Flink system is 3, the overall recommended parallelism of Flink jobs is 6, and the resource slot utilization rate is 70%, then the resource slot utilization rate after scaling up will be 100%.

[0170] For example, if the current parallelism is 3 and the recommended parallelism for Flink jobs is 3, then for Kalfa jobs, there is no need to expand or shrink the size.

[0171] In summary, the steps for adjusting the capacity of a Flink job based on the relationship between the first and third parallelism include:

[0172] If the first parallelism is greater than the third parallelism, the Flink job will be scaled down.

[0173] If the first parallelism is less than the third parallelism, the Flink job will be scaled up.

[0174] If the first parallelism is equal to the third parallelism, then the Flink job is executed according to the current resource slot capacity.

[0175] The process of scaling down or scaling up a Flink job includes:

[0176] The resource slot capacity is determined based on the ratio of the first parallelism to the third parallelism.

[0177] Referring to Figures 4(a) and 4(b), Figure 4(a) shows the resource load variation under static configuration, where the dashed line represents the load of the Flink system and the solid line represents the resource configuration of Flink. Referring to Figure 4(a), it can be seen that under static configuration, the Flink resource configuration remains constant. In contrast, Figure 4(b) shows that, with elastic scaling, the resource configuration changes with the load.

[0178] As mentioned above, in order to obtain the aforementioned first performance utilization rate, it is necessary to calculate the performance value of the vertex. Specifically, the performance value of the root node can be determined in the following way:

[0179] Continuing with the example of the Kalfa root node, assuming there are 10 downstream nodes under the Kalfa root node, we will first calculate the performance values ​​of these 10 downstream nodes.

[0180] Suppose that a vertex downstream of the Kalfa root node has multiple tasks.

[0181] First, calculate the performance value for each task, using the following formula:

[0182]

[0183] k — Performance value of the first target execution unit;

[0184] busyTimeMsPerSecond task —The time per second during which the first target execution unit is busy;

[0185] consumption task —The consumption rate of the performance value of the first target execution unit;

[0186] It should be noted that the first target execution unit corresponds to any one of the multiple tasks under a certain vertex.

[0187] Based on this, for any Vertex, the performance value of all tasks can be calculated using the above formula.

[0188] Here, the time the first target execution unit is busy per second is calculated in milliseconds, so the value is at most 1000.

[0189] Specifically, the consumption rate of each task can be obtained in real time through the Flink system's API, and k = throughput. task ;

[0190] Please refer to Figure 5 The busyTimeMsPerSecond of each task task The busy time per second is different, meaning that the busy level of each task is not consistent. Among many tasks, there will always be one task whose busy time per second reaches 1000 milliseconds, which is the bottleneck. Here, the task that reaches the bottleneck first is selected, and the performance value of the vertex is determined by the performance value of the task that reaches the bottleneck first, the consumption rate of the vertex, and the consumption rate of the task that reaches the bottleneck first.

[0191] It should be noted that when the first target execution unit is busy for 1000 seconds per second, it means that the first target execution unit has reached its bottleneck.

[0192] Right now

[0193] —Consumption rate of the second target execution unit;

[0194] —Performance values ​​of the second target execution unit;

[0195] consumptionRate vertex —Consumption rate of the target vertex;

[0196] It should be noted that the second target execution unit is the task that reaches the bottleneck first.

[0197] The reason for determining the performance value of a vertex by identifying the task that first reaches the bottleneck is as follows: If a task reaches the bottleneck at a vertex, the backpressure mechanism of the Flink system will push the new traffic upstream layer by layer, resulting in a backlog at the source node, while the traffic of the vertex where the task is located and its upstream nodes, as well as the root node of the source end, will no longer increase.

[0198] This explains why Vertex performance is determined by the task that reaches its bottleneck first.

[0199] That is: calculating the performance value of at least one target vertex includes:

[0200] Select at least one first target execution unit and calculate the performance value of at least one first target execution unit, wherein the first target execution unit is the execution unit of any target vertex among at least one target vertex;

[0201] A second target execution unit is selected from at least one first target execution unit, wherein the second target execution unit is the execution unit that reaches the bottleneck first among at least one first target execution unit;

[0202] The performance value of any target vertex among at least one target vertex is determined based on the performance value of the second target execution unit, the consumption rate of the second target execution unit, and the consumption rate of any target vertex among at least one target vertex. The performance value of the second target execution unit exists within the performance value of at least one first target execution unit.

[0203] To elaborate further, assuming we want to calculate the performance value of a certain Vertex, the first target execution unit corresponds to any task of that Vertex, while the second target execution unit corresponds to the task of that Vertex that reaches the bottleneck first.

[0204] When the performance of a Vertex reaches its bottleneck, Flink's backpressure will cause all upstream Vertexes to stop increasing, including the upstream root node Source.

[0205] Since the data distribution is stable over a short period, the convergence ratio of each vertex also remains relatively stable; that is, the ratio of outflow to inflow at each vertex is stable. Based on this, for any vertex, the resulting source performance value is:

[0206]

[0207] throughout source (vertex) — Performance value determined based on the target vertex;

[0208] consumptionRate source — Root node consumption rate;

[0209] consumptionRate vertex —Consumption rate of the target vertex;

[0210] throughout vertex —Performance value of the target vertex.

[0211] At this point, the corresponding root node performance value can be determined based on the 10 downstream nodes, and the minimum value among these 10 performance values ​​can be selected as the root node performance value.

[0212] That is: throughout source =minthroughout source (vertex);

[0213] throughout source (vertex) — A performance value determined based on at least one target vertex;

[0214] throughpout source —The performance value of any node in at least one root node;

[0215] Given the aforementioned Kalfa root node with 10 downstream nodes, the performance value determined based on at least one target vertex corresponds to the node performance value determined based on the 10 downstream nodes of the Kalfa root node.

[0216] It should be noted that, in at least one root node, the performance value of any one of the root nodes is the same as the performance value of the corresponding Kalfa root node.

[0217] In summary, the steps for calculating the performance value of any one of the at least one root nodes include:

[0218] The steps for calculating the performance value of any one of the at least one root nodes include:

[0219] Calculate the performance value of at least one target vertex;

[0220] Based on the performance value of at least one target vertex, the consumption rate of at least one target vertex, and the consumption rate of any one of the at least one root nodes, calculate the performance value of at least one root node. The target vertex is the downstream node of any one of the at least one root nodes.

[0221] Select at least one root node from at least one root node performance value, wherein the performance value of any one root node is the minimum value among the performance values ​​of at least one target vertex.

[0222] Based on this, we can calculate the performance value for any Source.

[0223] It should be noted that the method for obtaining performance values ​​in this application is only one possible implementation and is not intended to limit this application.

[0224] The following describes the methods for calculating the consumption rate and production rate in this application.

[0225] In summary, in this application, each Vertex has multiple Tasks. Assuming that for a given Vertex, N Tasks execute simultaneously, the consumption rate for any given Vertex is calculated as follows:

[0226]

[0227] consupmptionRate task —The consumption rate of the task;

[0228] consumptionRate vertex —Target vertex consumption rate;

[0229] Similarly, assuming N tasks are executed simultaneously under a certain root node, there are multiple execution units, and the consumptionRate will... source (Root node consumption rate) is the sum of the consumption rates of N execution units.

[0230] It should also be noted that the traffic of each execution unit, i.e. the consumption rate of each task, can be directly obtained through the API provided by the Flink system.

[0231] At this production rate, the following formula can be used in relevant technologies for calculation:

[0232] productionRate=consunptionRate+lagIncreaseRate;

[0233] consumption rate;

[0234] lagIncreaseRate — Backlog growth rate;

[0235] production rate;

[0236] The consumption rate and backlog growth rate can be obtained using the Flink system's API interface. For the calculation method of the consumption rate, please refer to the previous section, which will not be repeated here.

[0237] Regarding the backlog growth rate, which is the difference between two consecutive backlog data volumes, the backlog growth rate of the root node can be calculated as follows:

[0238]

[0239] t—the time when the indicator was last collected;

[0240] Δt—The time difference between the current time of data collection and the time of the last data collection;

[0241] lagIncreaseRate source — Root node backlog growth rate.

[0242] Based on this, the production rate of the root node can be calculated:

[0243] production rate source =consumptionRate source +lagIncreaseRate source ;

[0244] production rate source —The production rate of the root node;

[0245] lagIncreaseRate source —The backlog growth rate of the root node;

[0246] consumptionRate source — The consumption rate of the root node.

[0247] The production rate and consumption rate of the root node, as well as the backlog growth rate of the root node, are detailed above and will not be repeated here.

[0248] Based on this, the production rate of any one of the at least one root nodes can be calculated as follows:

[0249] Determine the consumption rate of any node in at least one root node, and the backlog growth rate of any node in at least one root node.

[0250] In at least one root node, the production rate of any one root node is the sum of the consumption rate of any one root node and the backlog growth rate of any one root node.

[0251] This application has the following beneficial effects:

[0252] 1) This application uses the performance utilization rate as an indicator to reflect the processing performance and workload of the Flink system. There is no need to consider other indicators such as CPU utilization. The system is more stable when expanding or shrinking.

[0253] 2) By using the performance utilization metric, the current parallelism is compared with the overall recommended parallelism of the Flink system, making scaling more accurate, improving system availability, reducing the number of adjustments, and avoiding resource waste.

[0254] Based on this, please refer to the embodiments provided in this application. Figure 6 , Figure 6 The Flink system architecture diagram provided in this application embodiment includes: an elastic scaler, an index collector, and a scheduler.

[0255] Assuming that the automatic tuning service of the Flink system is enabled, the metric collector and elastic scaler are started periodically, that is, they are started according to the collection cycle. For example, the metric collector and elastic scaler can be started once every minute. In this application, the metric collector is specifically responsible for calling the Flink API to collect metrics. The elastic scaler is used to calculate the recommended parallelism of the overall Flink system and determine whether to expand or shrink the system based on the relationship between the recommended parallelism and the current parallelism. The result is sent to the scheduler. The scheduler and related components (scheduling service) based on the recommended parallelism stop the old jobs and start the new jobs according to the new parallelism, specifically performing the expansion and shrinking operations.

[0256] For details on how to calculate the recommended parallelism, please refer to the previous explanation; it will not be repeated here.

[0257] Understandably, in this system, the data flow is as follows: the metric collector calls the Flink system's API to transmit the metrics to the elastic scaler, as shown by the dashed line, and the control flow is shown by the solid line.

[0258] For the specific process details of Flink Elastic Jobs, please refer to [link / reference needed]. Figure 7 First, for a specific Flink job, the production rate and performance utilization of each root node are calculated. Then, the recommended parallelism of each root node is calculated based on the correspondence between its current parallelism, performance utilization, and target utilization. The minimum value among the recommended parallelisms of each root node is selected as the recommended parallelism of the entire Flink system. The current parallelism of the Flink job is compared with the recommended parallelism of the entire Flink system. If the current parallelism of the Flink job is greater than the recommended parallelism of the entire Flink system, the system is scaled down. If the current parallelism of the Flink job is less than the recommended parallelism of the entire Flink system, the system is scaled up. If the current parallelism of the Flink job is equal to the recommended parallelism of the entire Flink system, the system capacity is not adjusted.

[0259] Please refer to Figure 8 , Figure 8 A flowchart of a Flink job elastic scaling method provided in this application embodiment specifically includes the following steps:

[0260] S801: Calculate the first performance utilization rate based on the performance value of any node in at least one root node and the production rate of any node in at least one root node; the first performance utilization rate is: the performance utilization rate of any vertex in at least one root node under the current Flink job;

[0261] S802: Determine at least one second parallelism based on at least one first performance utilization, a first parallelism, and a target performance utilization. The first parallelism is the current parallelism of the Flink job, and the second parallelism is the recommended parallelism of any vertex of at least one root node in the current Flink job.

[0262] S803: Determine the third parallelism based on at least one second parallelism, where the third parallelism is the maximum parallelism present in at least one second parallelism;

[0263] S804: Adjusts the capacity of a Flink job based on the relationship between the first and third parallelism.

[0264] One possible approach is to adjust the capacity of a Flink job based on the relationship between the first and third degrees of parallelism, including the following steps:

[0265] If the first parallelism is greater than the third parallelism, the Flink job will be scaled down.

[0266] If the first parallelism is less than the third parallelism, the Flink job will be scaled up.

[0267] If the first parallelism is equal to the third parallelism, then the Flink job is executed according to the current resource slot capacity.

[0268] One possible approach is to shrink or expand a Flink job, which involves the following steps:

[0269] The resource slot capacity is determined based on the ratio of the first parallelism to the third parallelism.

[0270] One possible approach is to further include, before calculating the first performance utilization rate based on the performance value of any node in at least one root node and the production rate of any node in at least one root node:

[0271] Calculate the performance value of any one of the at least one root nodes.

[0272] One possible approach is to compute the performance value of any one of the at least one root nodes by including the following steps:

[0273] Calculate the performance value of at least one target vertex;

[0274] Based on the performance value of at least one target vertex, the consumption rate of at least one target vertex, and the consumption rate of any one of the at least one root nodes, calculate the performance value of at least one root node. The target vertex is the downstream node of any one of the at least one root nodes.

[0275] Select at least one root node from at least one root node performance value, wherein the performance value of any one root node is the minimum value among the performance values ​​of at least one target vertex.

[0276] One possible approach is to calculate the performance value of at least one target vertex by including:

[0277] Select at least one first target execution unit and calculate the performance value of at least one first target execution unit, wherein the first target execution unit is the execution unit of any target vertex among at least one target vertex;

[0278] A second target execution unit is selected from at least one first target execution unit, wherein the second target execution unit is the execution unit that reaches the bottleneck first among at least one first target execution unit;

[0279] The performance value of any target vertex among at least one target vertex is determined based on the performance value of the second target execution unit, the consumption rate of the second target execution unit, and the consumption rate of any target vertex among at least one target vertex. The performance value of the second target execution unit exists within the performance value of at least one first target execution unit.

[0280] One possible approach is to calculate the first performance utilization rate based on the performance value of any node in at least one root node and the production rate of any node in at least one root node.

[0281] The performance utilization of any node in at least one root node under the current Flink job is calculated using the following formula:

[0282]

[0283] utilization source —First performance utilization;

[0284] production rate source —The production rate of the root node;

[0285] throughout source —Root node performance value.

[0286] One possible approach is to calculate the second parallelism using the following formula in the step of determining the second parallelism based on the first performance utilization, the first parallelism, and the target performance utilization:

[0287]

[0288] parallelism recommend —Second degree of parallelism;

[0289] utilization source —First performance utilization;

[0290] utilization target —Target performance utilization;

[0291] parallelism current —First degree of parallelism.

[0292] One possible approach is to calculate the production rate of any one of the at least one root node as follows:

[0293] Determine the consumption rate of any node in at least one root node, and the backlog growth rate of any node in at least one root node.

[0294] In at least one root node, the production rate of any one root node is the sum of the consumption rate of any one root node and the backlog growth rate of any one root node.

[0295] One possible approach is to perform the following steps in the next detection cycle:

[0296] The second performance utilization rate is calculated based on the performance value of any node in at least one root node and the production rate of any node in at least one root node; the second performance utilization rate is: the performance utilization rate of any vertex in at least one root node in the next cycle of the current detection cycle.

[0297] Determine whether the difference between the second performance utilization rate and the first performance utilization rate exceeds a threshold.

[0298] If so, the first performance utilization rate is updated based on the second performance utilization rate, and the step of determining at least one second parallelism rate based on the at least one first performance utilization rate, the first parallelism rate, and the target performance utilization rate is performed.

[0299] If not, perform the step of determining at least one second parallelism based on the at least one first performance utilization, the first parallelism, and the target performance utilization.

[0300] Please refer to the figure below for an embodiment of the Flink job elastic scaling device provided in this application, including:

[0301] First calculation module: used to calculate the first performance utilization rate based on the performance value of any node in at least one root node and the production rate of any node in at least one root node; the first performance utilization rate is: the performance utilization rate of any vertex in at least one root node under the current Flink job;

[0302] The second computation module is used to determine at least one second parallelism based on at least one first performance utilization, a first parallelism, and a target performance utilization. The first parallelism is the current parallelism of the Flink job, and the second parallelism is the recommended parallelism of any vertex of at least one root node in the current Flink job.

[0303] Determining module: used to determine a third degree of parallelism based on at least one second degree of parallelism, wherein the third degree of parallelism is the maximum degree of parallelism that exists in at least one second degree of parallelism;

[0304] Adjustment module: Used to adjust the capacity of Flink jobs based on the relationship between the first parallelism and the third parallelism.

[0305] One possible approach is that the adjustment module is specifically used for:

[0306] If the first parallelism is greater than the third parallelism, the Flink job will be scaled down.

[0307] If the first parallelism is less than the third parallelism, the Flink job will be scaled up.

[0308] If the first parallelism is equal to the third parallelism, then the Flink job is executed according to the current resource slot capacity.

[0309] One possible approach is that the adjustment module is specifically used for:

[0310] The resource slot capacity is determined based on the ratio of the first parallelism to the third parallelism.

[0311] One possible approach is to also include: a third calculation module: used to calculate the performance value of any one of the at least one root nodes.

[0312] One possible approach is that the third module is specifically used for:

[0313] Calculate the performance value of at least one target vertex;

[0314] Based on the performance value of at least one target vertex, the consumption rate of at least one target vertex, and the consumption rate of any one of the at least one root nodes, calculate the performance value of at least one root node. The target vertex is the downstream node of any one of the at least one root nodes.

[0315] Select at least one root node from at least one root node performance value, wherein the performance value of any one root node is the minimum value among the performance values ​​of at least one target vertex.

[0316] One possible placement is that the third module is specifically used for:

[0317] Select at least one first target execution unit and calculate the performance value of at least one first target execution unit, wherein the first target execution unit is the execution unit of any target vertex among at least one target vertex;

[0318] A second target execution unit is selected from at least one first target execution unit, wherein the second target execution unit is the execution unit that reaches the bottleneck first among at least one first target execution unit;

[0319] The performance value of any target vertex among at least one target vertex is determined based on the performance value of the second target execution unit, the consumption rate of the second target execution unit, and the consumption rate of any target vertex among at least one target vertex. The performance value of the second target execution unit exists within the performance value of at least one first target execution unit.

[0320] One possible approach is for the first computation module to calculate the performance utilization of any node among at least one root node in the current Flink job using the following formula:

[0321]

[0322] utilization source —First performance utilization;

[0323] production rate source —The production rate of the root node;

[0324] throughout source —Root node performance value.

[0325] One possible approach is for the second computation module to calculate the second degree of parallelism using the following formula:

[0326]

[0327] parallelism recommend —Second degree of parallelism;

[0328] utilization source —First performance utilization;

[0329] utilization target —Target performance utilization;

[0330] parallelism current —First degree of parallelism.

[0331] One possible approach is to calculate the production rate of any one of the at least one root node as follows:

[0332] Determine the consumption rate of any node in at least one root node, and the backlog growth rate of any node in at least one root node.

[0333] In at least one root node, the production rate of any one root node is the sum of the consumption rate of any one root node and the backlog growth rate of any one root node.

[0334] Corresponding to the above embodiments, this application also provides an electronic device. Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. The electronic device 1000 may include a processor 1001, a memory 1002, and a communication unit 1003. These components communicate through one or more buses. Those skilled in the art will understand that the structure of the electronic device shown in the figure does not constitute a limitation on the embodiment of the present invention. It may be a bus topology or a star topology, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0335] The communication unit 1003 is used to establish a communication channel, enabling the electronic device to communicate with other devices. It receives user data from other devices or sends user data to other devices.

[0336] The processor 1001 serves as the control center of the electronic device, connecting various parts of the device via interfaces and lines. It executes software programs, instructions, and / or modules stored in the memory 1002, and accesses data stored in the memory to perform various functions and / or process data. The processor may be composed of integrated circuits (ICs), such as a single packaged IC or multiple packaged ICs with the same or different functions connected together. For example, the processor 801 may consist only of a central processing unit (CPU). In this embodiment, the CPU may have a single processing core or include multiple processing cores.

[0337] The memory 1002 is used to store the execution instructions of the processor 801. The memory 1002 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0338] When the execution instructions in memory 1002 are executed by processor 1001, the electronic device 1000 is able to perform operations. Figure 1 Some or all of the steps in the illustrated embodiments.

[0339] In a specific implementation, the present invention also provides a computer storage medium, wherein the computer storage medium may store a program, which, when executed, may include some or all of the steps in the various embodiments of the simulation scene generation method provided by the present invention. The storage medium may be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0340] In a specific implementation, the present invention also provides a computer program product, wherein the computer program product includes executable instructions, which, when executed on a computer, cause the computer to perform some or all of the steps in various embodiments of the simulation scene generation method provided by the present invention.

[0341] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0342] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, the device embodiments and terminal embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

Claims

1. A method for elastic scaling of Flink jobs, characterized in that, Perform the following steps within the current testing cycle: The first performance utilization rate is calculated based on the performance value of any node in at least one root node and the production rate of any node in at least one root node; the first performance utilization rate is: the performance utilization rate of any vertex in at least one root node under the current Flink job; The performance value is used to characterize the rate at which a Flink job can process the number of records or the amount of data; the first performance utilization rate is obtained by dividing the production rate by the performance value. At least one second parallelism is determined based on the at least one first performance utilization, the first parallelism, and the target performance utilization. The first parallelism is the current parallelism of the Flink job, and the second parallelism is the recommended parallelism of any vertex of at least one root node in the current Flink job. The second parallelism is obtained by dividing the product of the first performance utilization and the first parallelism by the target performance utilization. A third parallelism is determined based on the at least one second parallelism, wherein the third parallelism is the maximum parallelism present in the at least one second parallelism. The capacity of the Flink job is adjusted based on the relationship between the first parallelism and the third parallelism.

2. The method according to claim 1, characterized in that, The steps for adjusting the capacity of a Flink job based on the relationship between the first parallelism and the third parallelism include: If the first parallelism is greater than the third parallelism, then the Flink job is scaled down. If the first parallelism is less than the third parallelism, then the Flink job is scaled up. If the first parallelism is equal to the third parallelism, then the Flink job is executed according to the current resource slot capacity.

3. The method according to claim 1, characterized in that, The process of scaling down or scaling up a Flink job includes: The resource slot capacity is determined based on the ratio of the first parallelism to the third parallelism.

4. The method according to claim 1, characterized in that, Before calculating the first performance utilization rate based on the performance value of any one of the at least one root nodes and the production rate of any one of the at least one root nodes, the method further includes: Calculate the performance value of any one of the at least one root nodes.

5. The method according to claim 4, characterized in that, The step of calculating the performance value of any one of the at least one root nodes includes: Calculate the performance value of at least one target vertex; Based on the performance value of the at least one target vertex, the consumption rate of the at least one target vertex, and the consumption rate of any one of the at least one root nodes, the performance value of at least one root node is calculated, wherein the target vertex is a downstream node of any one of the at least one root nodes. From the performance values ​​of the at least one root node, the performance value of any one root node is the minimum value among the performance values ​​of the at least one target vertex.

6. The method according to claim 5, characterized in that, The calculation of the performance value of at least one target vertex includes: Select at least one first target execution unit and calculate the performance value of at least one first target execution unit, wherein the first target execution unit is the execution unit of any target vertex among the at least one target vertex; A second target execution unit is selected from the at least one first target execution unit, wherein the second target execution unit is the execution unit that reaches the bottleneck first among the at least one first target execution units; The performance value of any target vertex among at least one target vertex is determined based on the performance value of the second target execution unit, the consumption rate of the second target execution unit, and the consumption rate of any target vertex among at least one target vertex. The performance value of the second target execution unit exists within the performance value of at least one first target execution unit.

7. The method according to any one of claims 1 to 6, characterized in that, The production rate of any one of the at least one root nodes is calculated as follows: Determine the consumption rate of any node in at least one root node, and the backlog growth rate of any node in at least one root node. The production rate of any one of the at least one root nodes is the sum of the consumption rate of any one of the at least one root nodes and the backlog growth rate of any one of the at least one root nodes.

8. The method according to claim 1, further comprising: In the next cycle of the current detection cycle, perform the following steps: The second performance utilization rate is calculated based on the performance value of any node in at least one root node and the production rate of any node in at least one root node; the second performance utilization rate is: the performance utilization rate of any vertex in at least one root node in the next cycle of the current detection cycle. Determine whether the difference between the second performance utilization rate and the first performance utilization rate exceeds a threshold. If so, the first performance utilization rate is updated based on the second performance utilization rate, and the step of determining at least one second parallelism rate based on the at least one first performance utilization rate, the first parallelism rate, and the target performance utilization rate is performed. If not, perform the step of determining at least one second parallelism based on the at least one first performance utilization, the first parallelism, and the target performance utilization.

9. A Flink flexible telescopic device, characterized in that, include: First calculation module: used to calculate the first performance utilization rate based on the performance value of any one of the at least one root nodes and the production rate of any one of the at least one root nodes; The first performance utilization rate is: the performance utilization rate of any vertex in at least one root node under the current Flink job; The performance value is used to characterize the rate at which a Flink job can process the number of records or the amount of data; the first performance utilization rate is obtained by dividing the production rate by the performance value. The second calculation module is used to determine at least one second parallelism based on the at least one first performance utilization rate, the first parallelism, and the target performance utilization rate. The first parallelism is the current parallelism of the Flink job, and the second parallelism is the recommended parallelism of any vertex of at least one root node in the current Flink job. The second parallelism is obtained by dividing the product of the first performance utilization rate and the first parallelism by the target performance utilization rate. Determining module: used to determine a third parallelism based on the at least one second parallelism, wherein the third parallelism is the maximum parallelism present in the at least one second parallelism; Adjustment module: Used to adjust the capacity of Flink jobs based on the relationship between the first parallelism and the third parallelism.

10. An electronic device, characterized in that, include: At least one processor; as well as At least one memory communicatively connected to the processor, wherein: The memory stores program instructions that can be executed by the processor, which can invoke the program instructions to perform the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer instructions that cause the computer to perform the method as described in any one of claims 1 to 8.

Images