Method and device for adjusting Flink SQL operator parallelism
Patent Information
- Application Number
- CN202110731394.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-06-30
- Publication Date
- 2024-12-10
- Estimated Expiration
- 2041-06-30
Smart Images
Figure CN113535354B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of Internet technology, and in particular to a method and device for adjusting the parallelism of a Flink SQL operator. Background Art
[0002] Apache Flink is an open source computing platform for distributed data stream processing and batch processing. It can provide functions to support both stream processing and batch processing applications based on the same Flink runtime. Due to its advantages such as supporting high throughput, low latency, high-performance stream processing and supporting Exactly-once semantics for stateful computing, it is increasingly favored by developers.
[0003] As more and more businesses migrate to Flink (OCDP platform provides visual drag tasks and Flink SQL tasks), higher requirements are placed on the utilization of system resources. Since Flink SQL tasks are pure SQL and more convenient, more and more developers choose SQL to develop tasks. However, Flink SQL tasks cannot adjust the parallelism at the operator level. If the parallelism at the task level is set directly, resources will be wasted. Therefore, how to adjust the parallelism of each operator conveniently and efficiently is very necessary in actual business operations. Summary of the invention
[0004] The object of the present invention is to provide a method and device for adjusting the parallelism of Flink SQL operators, which can conveniently and efficiently adjust the parallelism of each operator.
[0005] In order to achieve the above object, a first aspect of the present invention provides a method for adjusting the parallelism of a Flink SQL operator, comprising:
[0006] Initialize the Flink SQL task and the configured parallelism parameters to each operator in the Flink engine task execution plan;
[0007] Processing the Flink SQL task according to the parallelism parameter, and detecting the back pressure value during the execution of the Flink SQL task in real time;
[0008] Issue warnings for operators whose back pressure values exceed the threshold, adjust the configuration of parallelism parameters, and restart the task execution plan.
[0009] Preferably, before the step of initializing the Flink SQL task and the configured parallelism parameters to each operator in the Flink engine task execution plan, the following steps are also included:
[0010] Before the Flink SQL task goes online, the developer configures the parallelism parameters of the Flink SQL task through a visual interface.
[0011] Preferably, the method for processing the Flink SQL task according to the parallelism parameter and detecting the back pressure value in real time during the execution of the Flink SQL task includes:
[0012] During the execution of the Flink SQL task, the back pressure value of the task during runtime is obtained in real time based on the monitoring view and Flink UI.
[0013] Preferably, the method for issuing an early warning for an operator whose back pressure value exceeds a threshold, adjusting the configuration of parallelism parameters and restarting the task execution plan includes:
[0014] Compare the real-time back pressure value of each operator with the corresponding threshold, and issue an early warning when any one or more operators exceed the threshold.
[0015] According to the warning result, the developer is reminded to reconfigure the parallelism parameters and restart the task execution plan.
[0016] Furthermore, the method of reminding the developer to reconfigure the parallelism parameter and restart the task execution plan according to the early warning result includes:
[0017] Get the task topology plan from the context instance of the execution environment and parse the parallelism parameters of each operator;
[0018] Re-adjust the parallelism logic of operators based on the parallelism parameters of each operator, and reconfigure the parallelism parameters;
[0019] The parallelism logic of the operator will be readjusted and the task execution plan will be restarted with the reconfigured parallelism parameters, and submitted to the Flink engine for execution.
[0020] Optionally, the method of reminding the developer to reconfigure the parallelism parameter includes:
[0021] The developer is remotely reminded to reconfigure the parallelism parameter through any one or more methods including APP push, SMS, phone call, and email.
[0022] Compared with the prior art, the method for adjusting the parallelism of Flink SQL operators provided by the present invention has the following beneficial effects:
[0023] In the method for adjusting the parallelism of Flink SQL operators provided by the present invention, the Flink SQL task and the configured parallelism parameters are first initialized to each operator in the Flink engine task execution plan, and then the Flink engine is used to process the Flink SQL task according to the parallelism parameters. During the processing, the back pressure value of the Flink SQL task processing needs to be detected in real time, and the back pressure value is visualized to the developer, and an early warning is issued for the operator whose back pressure value exceeds the threshold, so that the developer can adjust the configuration of the parallelism parameter in time through the visual UI, and save the configuration after the parallelism parameter adjustment of the operator is completed. The operator parallelism can be modified by restarting the task.
[0024] It can be seen that the present invention solves the problem of being unable to adjust operator parallelism parameters for Flink SQL tasks, and improves the ability and efficiency of real-time task processing. In addition, since the developer completes the adjustment on the UI interface, compared with the prior art solution of adjusting operator parallelism by writing code, it has the function of quick restart and enhances the ability of task processing data.
[0025] A second aspect of the present invention provides a device for adjusting the parallelism of a Flink SQL operator, which is applied to the method for adjusting the parallelism of a Flink SQL operator described in the above technical solution, and the device includes:
[0026] The task launch unit is used to initialize the Flink SQL task and the configured parallelism parameters to each operator in the Flink engine task execution plan;
[0027] A running detection unit is used to process the Flink SQL task according to the parallelism parameter and detect the back pressure value during the execution of the Flink SQL task in real time;
[0028] The processing unit is used to issue warnings for operators whose backpressure values exceed the threshold, adjust the configuration of parallelism parameters, and restart the task execution plan.
[0029] Preferably, it also includes:
[0030] The parameter configuration unit is used for the developer to configure the parallelism parameters of the Flink SQL task through a visual interface before the Flink SQL task goes online.
[0031] Preferably, the operation detection unit is further used to obtain the back pressure value of the task during execution in real time according to the monitoring view and Flink UI during the execution of the Flink SQL task.
[0032] Preferably, the processing unit comprises:
[0033] The threshold comparison module is used to compare the real-time back pressure value of each operator with the corresponding threshold value, and issue an early warning when any one or more operators exceed the threshold value;
[0034] The early warning reminder module is used to remind the developer to reconfigure the parallelism parameters and restart the task execution plan.
[0035] Compared with the prior art, the beneficial effects of the device for adjusting the parallelism of Flink SQL operators provided by the present invention are the same as the beneficial effects of the method for adjusting the parallelism of Flink SQL operators provided by the above technical solution, which will not be described in detail here.
[0036] A third aspect of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned method for adjusting the parallelism of Flink SQL operators are executed.
[0037] Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present invention are the same as the beneficial effects of the method for adjusting the parallelism of Flink SQL operators provided by the above technical solution, which will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:
[0039] Figure 1 A flowchart of a method for adjusting the parallelism of a Flink SQ operator in an embodiment of the present invention is shown;
[0040] Figure 2 Another flowchart of a method for adjusting the parallelism of a Flink SQL operator in an embodiment of the present invention is shown;
[0041] Figure 3 This is an interactive sequence diagram of executing the Flink SQL operator in an embodiment of the present invention. DETAILED DESCRIPTION
[0042] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.
[0043] Embodiment 1
[0044] See also Figure 1-Figure 3 This embodiment provides a method for adjusting the parallelism of a Flink SQL operator, including:
[0045] Initialize the Flink SQL task and the configured parallelism parameters to each operator in the Flink engine task execution plan; process the Flink SQL task according to the parallelism parameters, and detect the back pressure value during the execution of the Flink SQL task in real time; issue an early warning for operators whose back pressure values exceed the threshold, adjust the configuration of the parallelism parameters, and restart the task execution plan.
[0046] Considering that the Flink community source code does not provide a solution for modifying the parallelism of Flink SQL task operators, in order to solve the above problem, this embodiment provides a method for adjusting the parallelism of Flink SQL operators. First, the Flink SQL task and the configured parallelism parameters are initialized to each operator in the Flink engine task execution plan, and then the Flink engine is used to process the Flink SQL task according to the parallelism parameters. During the processing, the back pressure value of the Flink SQL task processing needs to be detected in real time, and the back pressure value is visualized to the developer. Tasks with long-term consumption accumulation are analyzed, and operators with back pressure values exceeding the threshold are warned, so that developers can adjust the configuration of the parallelism parameters in time through the visual UI. After the parallelism parameters of the operator are adjusted, they are saved, and the operator parallelism can be modified by restarting the task.
[0047] It can be seen that this embodiment solves the problem of being unable to adjust operator parallelism parameters for Flink SQL tasks, and improves the ability and efficiency of real-time task processing. In addition, since the developer completes the adjustment on the UI interface, compared with the prior art solution of adjusting operator parallelism by writing code, the solution provided by this embodiment can be quickly restarted to take effect, enhancing the ability of tasks to process data.
[0048] In the above embodiment, before the step of initializing the Flink SQL task and the configured parallelism parameters to each operator in the Flink engine task execution plan, the following steps are also included:
[0049] Before the Flink SQL task goes online, the developer configures the parallelism parameters of the Flink SQL task through a visual interface. Compared with the traditional way of writing code, adjusting the parallelism parameters through the visual UI interface has the characteristics of low operation threshold, high efficiency, convenience and flexibility.
[0050] In the above embodiment, the method for processing the Flink SQL task according to the parallelism parameter and detecting the back pressure value in real time during the execution of the Flink SQL task includes:
[0051] During the execution of the Flink SQL task, the back pressure value of the task is obtained in real time based on the monitoring view and Flink UI.
[0052] In the above embodiment, the method for issuing an early warning for an operator whose back pressure value exceeds a threshold, adjusting the configuration of parallelism parameters and restarting the task execution plan includes:
[0053] The real-time back pressure value of each operator is compared with the corresponding threshold, and an early warning is issued when any one or more operators exceed the threshold; based on the early warning result, the developer is reminded to reconfigure the parallelism parameter and restart the task execution plan.
[0054] In specific implementation, the back pressure value shows the resource consumption of operators during task processing. According to the warning rules set by the user, when the back pressure value of one or more operators exceeds the threshold, an early warning can be issued to the developer in a timely manner.
[0055] In the above embodiment, the method for reminding the developer to reconfigure the parallelism parameter and restart the task execution plan according to the warning result includes:
[0056] Get the task topology plan from the context instance of the execution environment and parse the parallelism parameters of each operator;
[0057] Re-adjust the parallelism logic of operators based on the parallelism parameters of each operator, and reconfigure the parallelism parameters;
[0058] The parallelism logic of the operator will be readjusted and the task execution plan will be restarted with the reconfigured parallelism parameters, and submitted to the Flink engine for execution.
[0059] In the above embodiment, the method for reminding the developer to reconfigure the parallelism parameter includes:
[0060] Remotely remind developers to reconfigure parallelism parameters through any one or more methods including APP push, SMS, phone call, and email.
[0061] It needs to be explained that the execution graph in Flink can be divided into four layers: StreamGraph, JobGraph, ExecutionGraph and physical execution graph.
[0062] StreamGraph (task execution plan graph): is the initial graph generated based on the code written by the developer through the Stream API, used to represent the topology of the program. In this embodiment, the operator parallelism parameters are modified at this layer through the UI.
[0063] JobGraph (task optimization graph): StreamGraph is optimized to generate JobGraph, a data structure submitted to JobManager. The main optimization is to chain multiple qualified nodes together as one node, which can reduce the serialization / deserialization / transmission consumption required for data flow between nodes.
[0064] ExecutionGraph: JobManager generates ExecutionGraph based on JobGraph. ExecutionGraph is a parallelized version of JobGraph and is the core data structure of the scheduling layer.
[0065] Physical execution graph: After the JobManager schedules the Job according to the ExecutionGraph, it deploys the Task on each TaskManager to form a "graph" to run the task.
[0066] This embodiment obtains the parallelism parameters of each operator in the task StreamGraph from the context of the execution environment (StreamExecutionEnvironment). And it is displayed on the front-end page in the form of an interface. Then parse the parallelism parameters of each operator Operator Level (operator level). After obtaining the operator parallelism parameters, re-implement a set of logic that can modify the operator parallelism, then reset the operator parallelism parameters, and finally regenerate the task execution plan (StreamGraph) with the modified operator parallelism logic and the corresponding execution environment. Then submit the newly generated execution plan to the Flink computing engine for execution. Specifically, by customizing the subclass of StreamExecutionEnvironment, load the configuration in the parent class. Get the task execution plan instance, parse the operator parallelism set in the execution plan, convert the operator object from the context environment instance ContextEnvironment of the execution environment, and then call the method of re-implementing the setting of parallelism setParallelism, and finally deliver the adjusted task execution plan to the Flink engine for execution. Developers can view each operator and its parallelism parameters of the task on the front-end page, adjust the operator parallelism based on the actual situation and related indicators, and submit it to the background. The background program rewrites it into Flink's execution plan. That is, the task execution plan graph is re-implemented, and the newly adjusted operator parallelism parameters are registered into the task execution plan graph. In this process, this solution re-implements the execute method in the StreamExecutionEnvironment class and StreamPlanEnvironment. When the task is submitted for execution again (the task execution plan is restarted), the adjusted Flink SQL task will be executed, thereby achieving the purpose of modifying the FlinkSQL task operator parallelism.
[0067] It can be seen that this embodiment realizes the modification of the parallelism of StreamGraph and provides a visual operation entry to modify the operator parallelism. In addition, this embodiment is also applicable to visualization tasks to reduce the workload of modifying the code and repackaging the visualization tasks.
[0068] In summary, this embodiment achieves the purpose of adjusting the parallelism of Flink SQL operators, and solves the problem that developers cannot modify the parallelism of Flink SQL task operators. At the same time, it also greatly reduces the workload of task developers in modifying the parallelism of Flink visualization task operators, and can also solve the problem of reduced consumption capacity of the entire task caused by consumption accumulation of a certain operator in a very short time.
[0069] Embodiment 2
[0070] This embodiment provides a device for adjusting the parallelism of a Flink SQL operator, including:
[0071] The task launch unit is used to initialize the Flink SQL task and the configured parallelism parameters to each operator in the Flink engine task execution plan;
[0072] A running detection unit is used to process the Flink SQL task according to the parallelism parameter and detect the back pressure value during the execution of the Flink SQL task in real time;
[0073] The processing unit is used to issue warnings for operators whose backpressure values exceed the threshold, adjust the configuration of parallelism parameters, and restart the task execution plan.
[0074] Preferably, it also includes:
[0075] The parameter configuration unit is used for the developer to configure the parallelism parameters of the Flink SQL task through a visual interface before the Flink SQL task goes online.
[0076] Preferably, the operation detection unit is further used to obtain the back pressure value of the task during execution in real time according to the monitoring view and Flink UI during the execution of the Flink SQL task.
[0077] Preferably, the processing unit comprises:
[0078] The threshold comparison module is used to compare the real-time back pressure value of each operator with the corresponding threshold value, and issue an early warning when any one or more operators exceed the threshold value;
[0079] The early warning reminder module is used to remind developers to obtain the task topology plan from the context instance in the execution environment according to the early warning results, parse the parallelism parameters of each operator, readjust the parallelism logic of the operator based on the parallelism parameters of each operator, and reconfigure the parallelism parameters. The task execution plan will be restarted with the readjusted operator parallelism logic and reconfigured parallelism parameters, and submitted to the Flink engine for execution.
[0080] Compared with the prior art, the beneficial effect of the device for adjusting the parallelism of Flink SQL operators provided in the embodiment of the present invention is the same as the beneficial effect of the method for adjusting the parallelism of Flink SQL operators provided in the above-mentioned embodiment 1, which will not be described in detail here.
[0081] Embodiment 3
[0082] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for adjusting the parallelism of Flink SQL operators are executed.
[0083] Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this embodiment are the same as the beneficial effects of the method for adjusting the parallelism of Flink SQL operators provided by the above technical solution, which will not be described in detail here.
[0084] A person skilled in the art can understand that all or part of the steps in the above-mentioned inventive method can be implemented by instructing related hardware through a program, and the above-mentioned program can be stored in a computer-readable storage medium. When the program is executed, it includes the steps of the above-mentioned embodiment method, and the storage medium can be: ROM / RAM, disk, optical disk, memory card, etc.
[0085] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.
Claims
1. A method for adjusting the parallelism of a Flink SQL operator, characterized in that: include: Initialize the Flink SQL task and the configured parallelism parameters to each operator in the Flink engine task execution plan; Processing the Flink SQL task according to the parallelism parameter, and detecting the back pressure value during the execution of the Flink SQL task in real time; Issue warnings for operators whose back pressure values exceed the threshold, adjust the configuration of parallelism parameters, and restart the task execution plan; Among them, adjusting the configuration of the parallelism parameter and restarting the task execution plan includes: obtaining the task topology plan from the context instance of the execution environment, and parsing the parallelism parameters of each operator; Re-adjust the parallelism logic of operators based on the parallelism parameters of each operator, and reconfigure the parallelism parameters; Restart the task execution plan with the readjusted operator parallelism logic and reconfigured parallelism parameters, and submit it to the Flink engine for execution. The method of readjusting the parallelism logic of the operator based on the parallelism parameters of each operator, reconfiguring the parallelism parameters, restarting the task execution plan with the readjusted parallelism logic of the operator and the reconfigured parallelism parameters, and submitting it to the Flink engine for execution includes: through a subclass of a custom execution environment, loading the configuration in the parent class, obtaining a task execution plan instance, parsing the operator parallelism set in the execution plan, converting the operator object from the context environment instance of the execution environment to obtain the operator parallelism parameters, reconfiguring the operator parallelism parameters, and registering the reconfigured operator parallelism parameters in the task execution plan graph to deliver the adjusted task execution plan to the Flink engine for execution.
2. The method according to claim 1, characterized in that Before the step of initializing the Flink SQL task and the configured parallelism parameters to each operator in the Flink engine task execution plan, it also includes: Before the Flink SQL task goes online, the developer configures the parallelism parameters of the Flink SQL task through a visual interface.
3. The method according to claim 1 or 2, characterized in that: The method for processing the FlinkSQL task according to the parallelism parameter and detecting the back pressure value in real time during the execution of the Flink SQL task includes: During the execution of the Flink SQL task, the back pressure value of the task during runtime is obtained in real time based on the monitoring view and Flink UI.
4. The method according to claim 1, characterized in that: Methods for issuing warnings for operators whose back pressure values exceed the threshold, adjusting the configuration of parallelism parameters, and restarting the task execution plan include: Compare the real-time back pressure value of each operator with the corresponding threshold, and issue an early warning when any one or more operators exceed the threshold. According to the warning result, the developer is reminded to reconfigure the parallelism parameters and restart the task execution plan.
5. A device for adjusting the parallelism of a Flink SQL operator, characterized in that: include: The task launch unit is used to initialize the Flink SQL task and the configured parallelism parameters to each operator in the Flink engine task execution plan; A running detection unit is used to process the Flink SQL task according to the parallelism parameter and detect the back pressure value during the execution of the Flink SQL task in real time; The processing unit is used to issue warnings for operators whose backpressure values exceed the threshold, adjust the configuration of parallelism parameters, and restart the task execution plan; The processing unit includes: a threshold comparison module, which is used to compare the real-time back pressure value of each operator with the corresponding threshold value, and issue an early warning when any one or more operators exceed the threshold value; An early warning reminder module is used to remind developers to obtain the task topology plan from the context instance of the execution environment according to the early warning results, parse the parallelism parameters of each operator, parse the parallelism parameters of each operator; readjust the parallelism logic of the operator based on the parallelism parameters of each operator, and reconfigure the parallelism parameters, restart the task execution plan with the readjusted operator parallelism logic and the reconfigured parallelism parameters, and submit it to the Flink engine for execution and operation. The readjustment of the operator parallelism logic based on the parallelism parameters of each operator and reconfiguration of the parallelism parameters, restarting the task execution plan with the readjusted operator parallelism logic and the reconfigured parallelism parameters, and submitting it to the Flink engine for execution and operation includes: through a subclass of a custom execution environment, loading the configuration in the parent class, obtaining a task execution plan instance, parsing the operator parallelism set in the execution plan, converting the operator object from the context environment instance of the execution environment to obtain the operator parallelism parameters, reconfiguring the operator parallelism parameters, and registering the reconfigured operator parallelism parameters in the task execution plan graph to deliver the adjusted task execution plan to the Flink engine for execution and operation.
6. The device according to claim 5, characterized in that Also includes: The parameter configuration unit is used for the developer to configure the parallelism parameters of the Flink SQL task through a visual interface before the Flink SQL task goes online.
7. The device according to claim 5 or 6, characterized in that The operation detection unit is also used to obtain the back pressure value of the task during execution in real time according to the monitoring view and Flink UI during the execution of the FlinkSQL task.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are executed.