Implementation method and system for realizing Flink adaptive parallelism dynamic capacity expansion and contraction based on index monitoring

By dynamically adjusting the parallelism of Flink jobs based on indicator monitoring, the problem of untimely data processing and data accumulation caused by static parallelism of Flink jobs is solved, and adaptive resource elastic expansion is achieved, avoiding abnormal job failures.

CN119003218AInactive Publication Date: 2024-11-22ZHEJIANG SHUXIN NETWORK CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411043098.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-22
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The parallelism of Flink jobs is static during their life cycle, resulting in data processing not being timely during restart, and it cannot be automatically expanded according to the data flow situation in time, resulting in data accumulation and job abnormal failure.

Method used

Through an indicator monitoring method, indicator data is collected, key data is calculated, such as the average value of busyTimeMsPerSecond, dynamically adjust the parallelism degree, and request or release resources based on the adjusted parallelism degree to achieve adaptive parallelism degree expansion.

Benefits of technology

The Flink program automatically increases parallelism during peak traffic, avoids data accumulation and job abnormal failure, and uses fewer resources to maintain operation during traffic trough periods, which improves the elastic expansion of resources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119003218A_ABST
    Figure CN119003218A_ABST
Patent Text Reader

Abstract

The invention discloses an implementation method and system for implementing Flink adaptive parallelism dynamic capacity expansion and contraction based on index monitoring, and belongs to the technical field of big data technology real-time computing. The method comprises the steps of collecting index data; obtaining key data according to the index data; according to the key data, updating the degree of parallelism; and applying or releasing resources according to the updated parallelism degree. According to the method, the problem that the job is automatically restarted (recovered from the latest checkpoint) after the parallelism degree of the program is automatically adjusted according to the change of the data source input rate or the performance of a working load is solved, and the self-adaptive resource elastic expansion and contraction of the flink program are realized; the parallelism degree can be automatically increased in the traffic peak period, peak data can be processed in time, and program abnormal failure caused by peak data accumulation can be effectively avoided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of big data technology real-time computing technology, and specifically to a method and system for realizing Flink adaptive parallelism dynamic expansion and contraction based on indicator monitoring. Background Art

[0002] As a high-throughput, low-latency distributed stream / batch computing engine, Flink has been widely used in the field of big data technology.

[0003] Currently, the parallelism of a Flink job is static during its lifecycle and is defined only once at submission time. Usually, a stream processing job can be stopped at a savepoint and then restarted with a different parallelism. Changing the parallelism of a Flink job in this way has the following two problems:

[0004] 1. Flink job-level restart is too rough and takes a long time. The data generated during the restart cannot be processed in time, and additional overhead is required to create a Savepoint.

[0005] 2. It cannot automatically restart according to the data flow situation in time. When the data flow increases suddenly, the parallelism cannot be expanded in time, resulting in data accumulation and inability to consume, which may cause abnormal job failure. Summary of the invention

[0006] The purpose of the present invention is to provide an efficient method and system for realizing Flink adaptive parallelism dynamic expansion and contraction based on indicator monitoring.

[0007] In order to solve the above technical problems, the present invention provides a method for realizing Flink adaptive parallelism dynamic expansion and contraction based on indicator monitoring, comprising the following steps:

[0008] Collect indicator data;

[0009] Get key data based on indicator data;

[0010] Update the parallelism based on critical data;

[0011] Apply for or release resources based on the updated parallelism.

[0012] Preferably, obtaining key data according to the indicator data specifically includes the following steps:

[0013] Obtain indicator data in real time according to the preset cycle and calculate key data.

[0014] Preferably, the indicator data includes the busyTimeMsPerSecond indicator, and the indicator data is acquired in real time according to a preset period, and the key data is calculated, which specifically includes the following steps:

[0015] Check the average value of the busyTimeMsPerSecond metric collected within 10 minutes every 5 minutes as key data.

[0016] Preferably, updating the degree of parallelism according to the critical data specifically includes the following steps:

[0017] When the average value of the busyTimeMsPerSecond metric is greater than the maximum metric threshold, increase the degree of parallelism;

[0018] When the average value of the busyTimeMsPerSecond metric is less than the minimum metric threshold, reduce the parallelism.

[0019] Preferably, the maximum indicator threshold is 800ms;

[0020] The minimum indicator threshold is 200ms.

[0021] Preferably, the increasing the degree of parallelism specifically comprises the following steps:

[0022] Expand the degree of parallelism by a factor of 2.

[0023] Preferably, reducing the degree of parallelism specifically comprises the following steps:

[0024] Reduce the degree of parallelism to 1 / 2 of the original.

[0025] Preferably, applying for or releasing resources according to the updated parallelism specifically includes the following steps:

[0026] When the degree of parallelism increases, apply for solt resources, restart the task and restore the data;

[0027] When the degree of parallelism decreases, the task is restarted and the data is restored to release idle solt resources.

[0028] Preferably, restoring data specifically includes the following steps:

[0029] Restore data from the latest checkpoint snapshot.

[0030] The present invention also provides a system for implementing Flink adaptive parallelism dynamic expansion and contraction based on indicator monitoring, including:

[0031] Indicator collection module, used to collect indicator data;

[0032] The calculation module is used to obtain key data based on the indicator data;

[0033] Graph parallelism update module, used to update parallelism based on key data;

[0034] The resource adjustment module is used to apply for or release resources according to the updated parallelism.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] The present invention can automatically adjust the parallelism of the program according to the amount of data flowing into the data source and then automatically restart the job (recover from the most recent checkpoint), thereby realizing adaptive resource elastic expansion and contraction of the Flink program: using fewer resources to maintain program operation during the traffic trough period, and automatically increasing the parallelism to process peak data in time during the traffic peak period, which can effectively avoid abnormal program failure caused by the accumulation of peak data. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.

[0038] Figure 1 It is a flow chart of a method for realizing Flink adaptive parallelism dynamic expansion and contraction based on indicator monitoring in the present invention;

[0039] Figure 2 This is the flow chart of k8s creating resources;

[0040] Figure 3 This is a test link diagram. DETAILED DESCRIPTION

[0041] Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present invention, so the present invention is not limited to the specific implementation disclosed below.

[0042] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0043] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0044] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0045] The present invention provides a method for realizing Flink adaptive parallelism dynamic expansion and contraction based on indicator monitoring, comprising the following steps:

[0046] Collect indicator data;

[0047] Get key data based on indicator data;

[0048] Update the parallelism based on critical data;

[0049] Apply for or release resources based on the updated parallelism.

[0050] Preferably, obtaining key data according to the indicator data specifically includes the following steps:

[0051] Obtain indicator data in real time according to the preset cycle and calculate key data.

[0052] Preferably, the indicator data includes the busyTimeMsPerSecond indicator, and the indicator data is acquired in real time according to a preset period, and the key data is calculated, which specifically includes the following steps:

[0053] Check the average value of the busyTimeMsPerSecond metric collected within 10 minutes every 5 minutes as key data.

[0054] Preferably, updating the degree of parallelism according to the critical data specifically includes the following steps:

[0055] When the average value of the busyTimeMsPerSecond metric is greater than the maximum metric threshold, increase the degree of parallelism;

[0056] When the average value of the busyTimeMsPerSecond metric is less than the minimum metric threshold, reduce the parallelism.

[0057] Preferably, the maximum indicator threshold is 800ms;

[0058] The minimum indicator threshold is 200ms.

[0059] Preferably, the increasing the degree of parallelism specifically comprises the following steps:

[0060] The parallelism is doubled.

[0061] Preferably, reducing the degree of parallelism specifically comprises the following steps:

[0062] Reduce the degree of parallelism to 1 / 2 of the original.

[0063] Preferably, applying for or releasing resources according to the updated parallelism specifically includes the following steps:

[0064] When the degree of parallelism increases, apply for solt resources, restart the task and restore the data;

[0065] When the degree of parallelism decreases, the task is restarted and the data is restored to release idle solt resources.

[0066] Preferably, restoring data specifically includes the following steps:

[0067] Restore data from the latest checkpoint snapshot.

[0068] The present invention also provides a system for implementing Flink adaptive parallelism dynamic expansion and contraction based on indicator monitoring, including:

[0069] Indicator collection module, used to collect indicator data;

[0070] The calculation module is used to obtain key data based on the indicator data;

[0071] Graph parallelism update module, used to update parallelism based on key data;

[0072] The resource adjustment module is used to apply for or release resources according to the updated parallelism.

[0073] The present invention solves the problem of being able to automatically adjust the parallelism of the program according to the change of data source input rate or workload performance and then automatically restart the job (recover from the most recent checkpoint), thereby realizing the adaptive resource elastic expansion and contraction of the Flink program: using fewer resources to maintain program operation during the traffic trough period, and automatically increasing the parallelism to process peak data in time during the traffic peak period, which can effectively avoid abnormal program failure caused by the accumulation of peak data.

[0074] In order to better illustrate the technical effect of the present invention, the present invention provides the following specific embodiments to illustrate the above technical process:

[0075] Example 1: A method for implementing Flink's adaptive parallelism dynamic expansion and contraction based on indicator monitoring, such as Figure 1 As shown, the following steps are included:

[0076] (1) Real-time collection of key indicators;

[0077] The key indicators include the following two indicators:

[0078] Indicator 1: The maximum busytime indicator of all subtasks at the same time;

[0079] Indicator 2: Maximum backpressure indicator of all subtasks at the same time

[0080] (2) Calculate the indicators within the inspection cycle and determine whether the graph parallelism needs to be updated;

[0081] (3) Update graph parallelism;

[0082] (4) Restart the job and restore the data from the most recent checkpoint snapshot.

[0083] (5) Apply for or release idle solt.

[0084] This embodiment also provides a Flink adaptive scheduler, including:

[0085] The indicator collection module is used to collect the latest key indicators and cache the indicator data within the most recent inspection cycle;

[0086] Graph parallelism update module: polls and checks the indicator data within the inspection cycle to determine whether the parallelism needs to be updated (increase / decrease);

[0087] Job restart module: When the parallelism increases, solt resources are applied for first, and then the job is restarted and restored from the latest checkpoint. When the parallelism decreases, the job is restarted and restored from the latest checkpoint. After the restart, the idle solt is released.

[0088] Example 1: A method for implementing Flink's adaptive parallelism and dynamic expansion and contraction based on indicator monitoring:

[0089] The process of creating resources in k8s is as follows: Figure 2 As shown, the following steps are included:

[0090] Get resource application requests through K8s;

[0091] K8s creates a job manager, Job Master, based on resource application requests;

[0092] Start the job manager job Master through the resource manager Resource Manager;

[0093] The job manager Job Master sends a solt resource application request to the resource manager Resource Manager based on the resource application request;

[0094] The Resource Manager sends a resource creation request to K8s based on the solt resource application request;

[0095] K8s creates a pod unit based on the resource creation request;

[0096] Register slot resources with the resource manager Resource Manager through the pod unit;

[0097] Request slot resource information from the pod unit through the resource manager Resource Manage;

[0098] Return slot resource information to the job manager job Master through the pod unit;

[0099] The job manager Job Master executes jobs based on slot resource information.

[0100] Test link such as Figure 3 As shown, the test scheme adopted is as follows:

[0101] Kakfa Producer Generator customizes the data generation script and writes data to the Kafka topic as the producer Kafka Source. Flink on k8s reads the data in the corresponding topic as a consumer and performs simple processing. Kafka Sink controls the amount of data generated by modifying the parameters of the data generation script, thereby forming data peaks and troughs.

[0102] When the amount of data increases, the busyTimeMsPerSecond indicator of the flink job (the time that the subtask is busy performing some actual work) will increase. The job will automatically poll and check (by default, every 5 minutes) the average value of busyTimeMsPerSecond in the recent period (by default, 10 minutes). When it is greater than 0.8 (that is, the average value of busyTimeMsPerSecond reaches 800ms), the job will increase the parallelism by itself (expand to 2 times the original) and apply for a new slot from the jobmanager (the job will calculate the slot required by the program based on the parallelism). If the number of free slots in the current cluster is insufficient, the jobmanager will automatically apply to register and start the new taskmanager required. After the slot application is successful, the job restart will be triggered (it will be restored from the most recent checkpoint point).

[0103] When data consumption is completed, data production is at a low point, and the busyTimeMsPerSecond indicator of the flink job will gradually decrease to a low level. The job will automatically poll and check (every 5 minutes by default) the average value of busyTimeMsPerSecond in the recent period (10 minutes by default). When it is less than 0.2 (that is, the average value of busyTimeMsPerSecond is less than 200ms), the job will automatically reduce the parallelism (reduced to 1 / 2 of the original) and trigger the job restart (will be restored from the most recent checkpoint). After the job is successfully restarted, the redundant slots will be released. At this time, if all the slots of the taskmanager are free, the jobmanager will cancel the taskmanager.

[0104] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules, modules or units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units, modules or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0105] The units may or may not be physically separated, and the components displayed as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0106] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0107] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part, and / or installed from a removable medium. When the computer program is executed by the central processing unit (CPU), the above-mentioned functions defined in the method of the present invention are executed. It should be noted that the above-mentioned computer-readable medium of the present invention can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of an electrical, magnetic, optical, electromagnetic, infrared segment, or semiconductor, or any combination of the above.

[0108] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0109] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for implementing Flink's adaptive parallelism dynamic expansion and contraction based on indicator monitoring, characterized in that: The following steps are involved: Collect indicator data; Get key data based on indicator data; Update the parallelism based on critical data; Apply for or release resources based on the updated parallelism.

2. According to the method for realizing Flink adaptive parallelism dynamic expansion and contraction based on indicator monitoring according to claim 1, it is characterized in that: According to the indicator data, key data is obtained, which specifically includes the following steps: Obtain indicator data in real time according to the preset cycle and calculate key data.

3. According to the method for realizing Flink adaptive parallelism dynamic expansion and contraction based on indicator monitoring according to claim 2, it is characterized in that: The indicator data includes the busyTimeMsPerSecond indicator. The indicator data is obtained in real time according to a preset period, and the key data is calculated, which specifically includes the following steps: Check the average value of the busyTimeMsPerSecond metric collected within 10 minutes every 5 minutes as key data.

4. According to claim 3, the method for realizing Flink adaptive parallelism dynamic expansion and contraction based on indicator monitoring is characterized in that: Update the degree of parallelism based on key data, including the following steps: When the average value of the busyTimeMsPerSecond metric is greater than the maximum metric threshold, increase the degree of parallelism; When the average value of the busyTimeMsPerSecond metric is less than the minimum metric threshold, reduce the parallelism.

5. The method for realizing Flink adaptive parallelism dynamic expansion and contraction based on indicator monitoring according to claim 4 is characterized in that: The maximum indicator threshold is 800ms; The minimum indicator threshold is 200ms.

6. The method for realizing Flink adaptive parallelism dynamic expansion and contraction based on indicator monitoring according to claim 5 is characterized in that: The increasing degree of parallelism specifically comprises the following steps: Expand the degree of parallelism by a factor of 2.

7. The method for realizing Flink adaptive parallelism dynamic expansion and contraction based on indicator monitoring according to claim 6 is characterized in that: The reducing degree of parallelism specifically comprises the following steps: Reduce the degree of parallelism to 1 / 2 of the original.

8. The method for realizing Flink adaptive parallelism dynamic expansion and contraction based on indicator monitoring according to claim 7 is characterized in that: According to the updated parallelism, resources are applied or released, which includes the following steps: When the degree of parallelism increases, apply for solt resources, restart the task and restore the data; When the degree of parallelism decreases, the task is restarted and the data is restored to release idle solt resources.

9. The method for realizing Flink adaptive parallelism dynamic expansion and contraction based on indicator monitoring according to claim 8 is characterized in that: Restoring data includes the following steps: Restore data from the latest checkpoint snapshot.

10. A system for implementing Flink's adaptive parallelism dynamic expansion and contraction based on indicator monitoring, used to implement the method for implementing Flink's adaptive parallelism dynamic expansion and contraction based on indicator monitoring as described in any one of claims 1 to 9, characterized in that: include: Indicator collection module, used to collect indicator data; The calculation module is used to obtain key data based on the indicator data; Graph parallelism update module, used to update parallelism based on key data; The resource adjustment module is used to apply for or release resources according to the updated parallelism.

Citation Information

Patent Citations

  • Streaming computing job processing method and device

    CN113672396A

  • System and method for realizing task resource elastic scaling based on flink task index monitoring

    CN115328667A