Adaptive adjustment method and device for Flink operation, equipment and storage medium

Abstract

The invention discloses an adaptive adjustment method and device for Flink operation, equipment and a storage medium, and relates to the technical field of computers, the method comprises the following steps: in response to an input data stream entering a time window, counting the total number of input stream data in the time window; in response to the fact that the total data amount is larger than the preset bearing capacity, whether the current operator parallelism degree meets the processing requirement of the input stream data or not is judged based on the adaptation relation between the total data amount and the operator parallelism degree so as to determine whether the current operator parallelism degree needs to be adjusted or not; in response to the condition that the parallelism degree of the current operator needs to be adjusted, determining the parallelism degree of a target operator based on the data processing amount and the total data amount of each operator in the Flink operation; and adjusting the current operator parallelism degree to be a target operator parallelism degree, and carrying out parallel processing on the input stream data based on the target operator parallelism degree. Self-adaptive adjustment and elastic expansion and contraction of the Flink job operator parallelism degree are realized, and the processing flexibility and the resource utilization rate of the Flink job in a load change scene without shutdown are remarkably improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, equipment and storage medium for adaptively adjusting a Flink job. Background Art

[0002] With the rapid development of financial technology, the amount of data in the financial industry has begun to surge. Various business scenarios generate a large amount of data all the time, including user financial transaction data, monitoring data, and log data. These data have one thing in common: they are generated from different data sources in real time and then transmitted to different downstream systems. In recent years, the industry has seen the rise of stateful stream computing solutions represented by Flink, which has efficient real-time data processing capabilities and can achieve continuous ingestion, calculation, and output of real-time data. It is widely used in scenarios such as real-time intelligent recommendations, real-time fraud monitoring, and real-time reporting.

[0003] In practical applications, Flink stream computing jobs are usually established and deployed in a Flink cluster. Each job applies for computing resources from the cluster on demand. Each job contains one to several operators, which are responsible for converting and processing data streams. Operators run in parallel, and the setting of operator parallelism and the scheduling between operators directly affect the computing speed of operators.

[0004] Usually, when a Flink job is started, the data processing logic and operator parallelism are determined. Generally, the resources allocated to the job are applied for according to the peak resources. However, the transaction volume is often uneven in time, resulting in resource waste or resource overload. When a problem occurs, the job needs to be stopped and the relevant logic and configuration need to be modified to adapt, which is very inflexible and prone to problems.

[0005] In view of this, how to improve the flexibility of Flink jobs has become an urgent problem to be solved. Summary of the invention

[0006] In view of this, the main purpose of the present application is to provide a method, device, equipment and storage medium for adaptive adjustment of Flink jobs, so as to realize adaptive adjustment of the operator parallelism of Flin jobs to improve the flexibility of Flink jobs.

[0007] In a first aspect, the present application provides a method for adaptively adjusting a Flink job, the method comprising:

[0008] In response to an input data stream entering the time window, counting the total number of data of the input stream data within the time window;

[0009] In response to the total amount of data being greater than the preset carrying capacity, based on the adaptation relationship between the total amount of data and the operator parallelism, determining whether the current operator parallelism meets the processing requirements of the input stream data, so as to determine whether the current operator parallelism needs to be adjusted;

[0010] In response to the need to adjust the current operator parallelism, the target operator parallelism is determined based on the data processing volume and total data volume of each operator in the Flink job;

[0011] Adjust the current operator parallelism to the target operator parallelism, and perform parallel processing of input stream data based on the target operator parallelism.

[0012] In some implementations of the first aspect of the present application, the method further includes:

[0013] In response to the input data stream entering the time window, the data ratio and data proportion weight of each data type in the input stream data within the time window are counted;

[0014] In response to the fact that a data ratio of at least one data type in the input stream data exceeds a corresponding preset planned distribution ratio of the data type, the data type exceeding the preset planned distribution ratio of the data type is used as a target data type;

[0015] In response to the data proportion weight of the target data type exceeding the corresponding preset value, the target data type is divided into at least two sub-data types, the data corresponding to the at least two sub-data types in the input stream data constitute the data corresponding to the target data type in the input stream data, and the data corresponding to the at least two sub-data types in the input stream data have no intersection.

[0016] In some implementations of the first aspect of the present application, the target data type is divided into at least two sub-data types, including:

[0017] The target data type is divided into at least two sub-data types by adding different fields to the type field of the target data type.

[0018] In some implementations of the first aspect of the present application, the method further includes:

[0019] In response to the fact that the sum of the data corresponding to the at least two sub-data types in the input stream data does not exceed the preset data type planned distribution ratio corresponding to the target data type, the at least two sub-data types are merged into the target data type.

[0020] In some implementations of the first aspect of the present application, the method further includes:

[0021] Obtain the operation status of operator resources, and based on the operation status of operator resources, determine whether the current operator parallelism meets the processing requirements of input stream data to determine whether the current operator parallelism needs to be adjusted.

[0022] In some implementations of the first aspect of the present application, the method further includes:

[0023] Obtain the data consumption delay, and based on the data consumption delay, determine whether the current operator parallelism meets the processing requirements of the input stream data to determine whether the current operator parallelism needs to be adjusted.

[0024] In some implementations of the first aspect of the present application, in response to the current operator parallelism needing to be adjusted, the method includes:

[0025] The settlement result is calculated based on the weights corresponding to the factors that need to be adjusted for the current operator parallelism. The factors include: adaptation relationship, operator resource operation status, and data consumption delay.

[0026] In response to the calculation result exceeding the preset value, it is determined whether the current operator parallelism needs to be adjusted.

[0027] A second aspect of the present application provides an adaptive adjustment device for a Flink job, the device comprising:

[0028] A data statistics module, used for counting the total number of input stream data in the time window in response to the input data stream entering the time window;

[0029] A condition determination module, for responding to the total amount of data being greater than a preset carrying capacity, based on the adaptation relationship between the total amount of data and the operator parallelism, determining whether the current operator parallelism meets the processing requirements of the input stream data, so as to determine whether the current operator parallelism needs to be adjusted;

[0030] The parallelism determination module is used to respond to the need to adjust the current operator parallelism and determine the target operator parallelism based on the data processing volume and total data volume of each operator in the Flink job;

[0031] The parallelism adjustment module is used to adjust the current operator parallelism to the target operator parallelism, and perform parallel processing of input stream data based on the target operator parallelism.

[0032] In a third aspect, the present application provides an adaptive adjustment device for a Flink job, the device comprising a memory and a processor, the processor being used to execute a program stored in the memory and to run any adaptive adjustment method for a Flink job provided in the first aspect of the present application.

[0033] In a fourth aspect, the present application provides a readable storage medium on which a computer program is stored. When the computer program is executed by a processor, each step of the adaptive adjustment method of any Flink job provided in the first aspect of the present application is implemented.

[0034] The technical solution provided by this application has the following beneficial effects:

[0035] In the technical solution provided in the present application, first, in response to the input data stream entering the time window, the total number of data of the input stream data in the time window is counted, and by dividing the data flow in a fixed time period, the total amount of input data in the time period is counted to reflect the load in the time window; then, in response to the total amount of data being greater than the preset carrying capacity, based on the adaptation relationship between the total amount of data and the operator parallelism, it is determined whether the current operator parallelism meets the processing requirements of the input stream data to determine whether the current operator parallelism needs to be adjusted, wherein if the total amount of data exceeds the preset carrying capacity, it is determined based on the adaptation relationship whether the current operator parallelism is sufficient to process the data flow in the current time window; thereafter, in response to the need to adjust the current operator parallelism, based on the data processing amount and the total amount of data of each operator in the Flink job, the target operator parallelism is determined, thereby realizing dynamic calculation of a reasonable target operator parallelism according to the real-time load; finally, the current operator parallelism is adjusted to the target operator parallelism, and parallel processing of the input stream data is performed based on the target operator parallelism, thereby realizing dynamic adjustment of the operator parallelism and reallocating resources and data streams to achieve efficient processing. Overall, this application realizes the adaptive adjustment and elastic scaling of the parallelism of Flink job operators by real-time statistical data traffic, dynamically judging the adaptation of operator parallelism and performing adjustments, which significantly improves the processing flexibility and resource utilization of Flink jobs without downtime in load change scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A flowchart of a method for adaptively adjusting a Flink job provided in an embodiment of the present application;

[0037] Figure 2 A flowchart of another method for adaptively adjusting a Flink job provided in an embodiment of the present application;

[0038] Figure 3 A flowchart of another method for adaptively adjusting a Flink job provided in an embodiment of the present application;

[0039] Figure 4 A schematic diagram of the overall process of a Flink job provided in an embodiment of the present application;

[0040] Figure 5A schematic diagram of the structure of a Flink job adaptive adjustment device provided in an embodiment of the present application;

[0041] Figure 6 A schematic diagram of the structure of a Flink job adaptive adjustment device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0043] Glossary:

[0044] Stream computing: Stream processing (SP) is a technology for processing large amounts of data streams. It is mainly used to process data streams in real time in order to quickly analyze and process data.

[0045] Flink: An open source stream computing framework for real-time processing and analysis of data streams. It is a stream computing framework with high performance, high fault tolerance, and high scalability.

[0046] Flink job: refers to a complete, compiled and optimized data processing flow, consisting of several interconnected operators.

[0047] Operator: An operator is a component that performs transformation operations on data streams. It is the basic unit used to process and transform data streams. Several operators can construct complex stream computing logic.

[0048] Parallelism: refers to the number of concurrently executed tasks that a Flink job is divided into in order to efficiently process the data stream. You can set the parallelism of all operators globally or for a specific operator individually.

[0049] Elastic scaling: Technology that automatically adjusts the number of computing resources to adapt to changes in application load requirements, improve application service quality and effective resource utilization efficiency.

[0050] In view of the technical problems mentioned in the background technology, the embodiments of the present application provide a method, apparatus, device and storage medium for adaptively adjusting a Flink job.

[0051] See also Figure 1 As shown, Figure 1 A flowchart of a method for adaptively adjusting a Flink job provided in an embodiment of the present application may specifically include the following steps:

[0052] S101: In response to an input data stream entering a time window, counting the total number of data of the input stream data within the time window.

[0053] In an embodiment of the present application, the input data stream refers to the data stream input from an external data source into a Flink job, which may be various types of data such as real-time logs, user business records, and user transaction records; and the time window is a mechanism in Fink data stream processing, which is used to divide the arriving data into time intervals, for example, 1 minute, 5 minutes, or 10 minutes as a window, wherein the total amount of input stream data in the time window directly reflects the load situation in the current time window.

[0054] S102: In response to the total amount of data being greater than the preset carrying capacity, based on the adaptation relationship between the total amount of data and the operator parallelism, determine whether the current operator parallelism meets the processing requirements of the input stream data to determine whether the current operator parallelism needs to be adjusted.

[0055] In an embodiment of the present application, the preset carrying capacity refers to the data carrying upper limit of a preset time window, and the preset carrying capacity can be calculated based on historical data, such as the average value of the total amount of data in a time window in multiple time periods, and the product obtained by multiplying it by 150% is used as the preset carrying capacity.

[0056] The preset carrying capacity is a safe range setting for the processing capability of the input data stream. For example, assuming that the current carrying capacity is to process 1,000 data items per second, this condition will be triggered if the total number of data items in the time window exceeds 1,000. When the total number of data items in the input data stream is greater than the preset carrying capacity, the current operator parallelism will be determined to meet the processing requirements of the input stream data based on the adaptation relationship between the total amount of data and the operator parallelism.

[0057] The above adaptation relationship describes how much operator parallelism is needed for different total amounts of data to ensure efficient processing. For example, if an operator can process 500 data items per second, then when processing 2,000 data items, at least 4 parallelisms are theoretically required; and operator parallelism refers to the number of parallel tasks running by the same operator. For example, the parallelism of operator A is 3, which means that there are 3 concurrent operator instances processing data at the same time. If the current operator parallelism is less than the required parallelism determined by the adaptation relationship, data back pressure will occur. Back pressure means that the data intake rate is greater than the data processing rate, and the operator cannot process the data in time, which may lead to memory overflow or other performance problems. Therefore, the operator parallelism needs to be adjusted at this time.

[0058] S103: In response to the need to adjust the current operator parallelism, a target operator parallelism is determined based on the data processing volume and the total amount of data of each operator in the Flink job.

[0059] The operators in the Flink job are the core components of data processing, performing operations such as filtering, aggregation, and sorting. Multiple operators can form a complete Flink job. In an embodiment of the present application, based on the current data load and the performance of the operator, the optimal operator parallelism that meets the data processing requirements of the input stream data is calculated as the target operator parallelism to ensure that the input stream data can be processed efficiently and the operators can be reasonably allocated. For example, assuming that the total amount of data in the current time window is 10,000, and the reasonable processing capacity of each operator is 2,000 / second, then 10,000 / 2,000=5, and 5 is the target operator parallelism.

[0060] It is understandable that data flow has the characteristic of being able to fluctuate at any time. The embodiments of the present application can adapt to burst traffic and avoid delay backlogs by dynamically determining the parallelism of the target operator, so that the Flink job can adapt to the ever-changing load and provide more stable real-time data processing capabilities.

[0061] In some implementations of the present application, the parallelism of the target operator can be further limited in combination with hardware resource restrictions. For example, assuming that the current server only allows 4 parallel tasks, even if the calculated parallelism of the target operator is 5, the maximum parallelism of the target operator is only allowed to be 4.

[0062] S104: Adjust the current operator parallelism to the target operator parallelism, and perform parallel processing of input stream data based on the target operator parallelism.

[0063] In an embodiment of the present application, the parallel processing configuration of the Flink job is dynamically modified according to the calculated target operator parallelism, thereby optimizing the real-time processing capability of the input stream data. After adjustment, the Flink job will reallocate data processing tasks according to the new operator parallelism to ensure that the input data stream can be processed efficiently. It is understandable that this adjustment process is dynamic, that is, the Flink job does not need to stop running, and the operator parallelism can be directly updated during the execution of the job.

[0064] exist Figure 1In the process shown, first, in response to the input data stream entering the time window, the total number of input stream data in the time window is counted, and the total amount of input data in the time period is counted by dividing the data flow in a fixed time period to reflect the load in the time window; then, in response to the total amount of data being greater than the preset carrying capacity, based on the adaptation relationship between the total amount of data and the operator parallelism, it is determined whether the current operator parallelism meets the processing requirements of the input stream data to determine whether the current operator parallelism needs to be adjusted, wherein if the total amount of data exceeds the preset carrying capacity, it is determined based on the adaptation relationship whether the current operator parallelism is sufficient to process the data flow in the current time window; then, in response to the need to adjust the current operator parallelism, based on the data processing volume and the total amount of data of each operator in the Flink job, the target operator parallelism is determined, thereby realizing the dynamic calculation of a reasonable target operator parallelism according to the real-time load; finally, the current operator parallelism is adjusted to the target operator parallelism, and the input stream data is processed in parallel based on the target operator parallelism, thereby realizing the dynamic adjustment of the operator parallelism and reallocating resources and data streams to achieve efficient processing. Overall, this application realizes adaptive adjustment and elastic scaling of the parallelism of Flink job operators by real-time statistical data traffic, dynamically judging the adaptation of operator parallelism and performing adjustments, which significantly improves the processing flexibility and resource utilization of Flink jobs without downtime in load change scenarios.

[0065] See also Figure 2 As shown, Figure 2 A flowchart of another method for adaptively adjusting a Flink job provided in an embodiment of the present application may specifically include the following steps:

[0066] S201: In response to an input data stream entering a time window, a data ratio and a data proportion weight of each data type in the input stream data within the time window are counted.

[0067] In an embodiment of the present application, the input stream data usually includes multiple data types, where the data ratio refers to the proportion of each data type in the total data volume within the time window, and the data proportion weight refers to the quantitative representation of the relative importance of different data types in the input stream data. For example, assuming there are three categories a, b and c, the unit time of the time window is calculated as 10 minutes, the length of a is 100, and there are 100 items; the length of b is 200, and there are 100 items; the length of c is 50, and there are 100 items. Then the data ratio of a is 100 / (100+100+100), and the weight ratio of a is (100*100) / ((100*100)+(200*100)+(50*100)).

[0068] S202: In response to the presence of at least one data type in the input stream data whose data ratio exceeds the corresponding preset data type planned distribution ratio, the data type exceeding the preset data type planned distribution ratio is used as the target data type.

[0069] In an embodiment of the present application, in the current time window, it is detected whether the proportion of certain data types exceeds the preset planned distribution ratio of the preset data types. If the actual proportion of a certain data type exceeds the planned range, it is marked as the target data type. Among them, the preset data type planned distribution ratio refers to the ideal proportion of each data type when the Flink job is running. This value can be determined by historical data analysis, manual setting or server performance monitoring as a benchmark for measuring whether the data distribution is ideal. The target data type refers to the data type whose actual data ratio exceeds the preset planned ratio in the current time window. For example, assume that the three types of data in the preset data type planned distribution ratio have the following proportions: data type a is 50%, and data type b is 30%. Data type c is 20%. When the proportion of data type a in the input stream data is 55%, data type a is used as the target data type.

[0070] It is understandable that the target data type is often the key factor causing data skew. Data skew refers to the uneven distribution of data in a job, which causes a certain operator to process much more data than other nodes, which will slow down the processing speed of the operator and even cause the entire job to fail.

[0071] In view of this, the present application balances resource allocation according to the different processing complexities and resource requirements of different data types. By setting a reasonable preset data type plan distribution ratio, it monitors the problem of certain data types occupying too many resources so that corresponding processing can be performed later, thereby avoiding the impact of data processing performance due to data skew.

[0072] S203: In response to the data proportion weight of the target data type exceeding the corresponding preset value, the target data type is divided into at least two sub-data types, the data corresponding to the at least two sub-data types in the input stream data constitute the data corresponding to the target data type in the input stream data, and the data corresponding to the at least two sub-data types in the input stream data have no intersection.

[0073] In an embodiment of the present application, in the input stream data, the data marked as the target data type is further split into smaller sub-data types, thereby alleviating the resource pressure caused by the excessive proportion of a certain type of data, and optimizing the performance and efficiency of the Flink job. Specifically, if the data proportion weight of the target data type is too high, that is, it exceeds the preset threshold corresponding to the target data type, there may be a situation where the current Flink job cannot efficiently process the input stream data. Therefore, by splitting the target data type, the large category of data is refined into smaller data types, so that operator resources can be more evenly allocated to process the input stream data.

[0074] The data corresponding to at least two sub-data types in the input stream data constitute the data corresponding to the target data type in the input stream data, and the data corresponding to at least two sub-data types in the input stream data have no intersection. For example, in the input stream data, the data of data type a can constitute a set A, and data types a1 and a2 are obtained by splitting data type a, and a1 and a2 correspond to subsets A1 and A2 respectively, then set A=A1∪A2, and A1 and A2 have no intersection, that is, there is no duplication between the data corresponding to the sub-data types.

[0075] In some implementations of the embodiments of the present application, the target data type is divided into at least two sub-data types, specifically including:

[0076] By adding different fields to the type field of the target data type, the target data type is divided into at least two sub-data types. For example, two fields are randomly added to the type field of the target data type. For example, if the type field of the target data type is type1, two sub-data types are obtained by adding fields, namely type11 and type12.

[0077] S204: In response to the fact that the sum of the data corresponding to the at least two sub-data types in the input stream data does not exceed the preset data type planned distribution ratio corresponding to the target data type, the at least two sub-data types are merged into the target data type.

[0078] In an embodiment of the present application, the sub-data types that have been split are dynamically monitored, and when conditions are met, they are re-merged into the original target data type to maintain reasonable resource allocation. Specifically, when the sum of the data proportions of two or more sub-data types in the input stream data is lower than the preset planned distribution ratio of the target data type, it is considered that there is no need to process these sub-data types separately, so they can be merged into the target data type. For example, in the preset data type planned distribution ratio, data type a is 50%, and data types a1 and a2 are obtained by splitting data type a, wherein, in the input stream data, the data proportion of a1 is 20%, and the data proportion of A2 is 20%, then (20%+20%) <50%, and sub-data types a1 and a2 can be merged back into data type a.

[0079] It can be seen that when the data flow decreases or the data distribution tends to be reasonable, the embodiment of the present application reduces resource overhead and avoids resource waste by merging sub-data types, while maintaining the processing efficiency and flexibility of the Flink job. This dynamic adjustment mechanism helps to improve the overall performance of the Flink job.

[0080] exist Figure 2 In the process shown, first, in response to the input data stream entering the time window, the data ratio and data proportion weight of each data type in the input stream data within the time window are counted; then, in response to the data ratio of at least one data type in the input stream data exceeding the corresponding preset data type planned distribution ratio, the data type exceeding the preset data type planned distribution ratio is used as the target data type; thereafter, in response to the data proportion weight of the target data type exceeding the corresponding preset value, the target data type is divided into at least two sub-data types, and the data corresponding to at least two sub-data types in the input stream data constitute the data corresponding to the target data type in the input stream data, and the data corresponding to at least two sub-data types in the input stream data have no intersection. Finally, in response to the data ratio of the sum of the data corresponding to at least two sub-data types in the input stream data not exceeding the preset data type planned distribution ratio corresponding to the target data type, at least two sub-data types are merged into the target data type; thereby effectively solving the data skew problem in Flink jobs.

[0081] See also Figure 3 As shown, Figure 3 Another adaptive adjustment method for Flink jobs provided in an embodiment of the present application describes in detail three different situations that can be used to determine whether the current operator parallelism needs to be adjusted, which may specifically include the following steps:

[0082] S301: In response to an input data stream entering a time window, counting the total number of data of the input stream data within the time window.

[0083] S302: In response to the total amount of data being greater than the preset carrying capacity, based on the adaptation relationship between the total amount of data and the operator parallelism, determine whether the current operator parallelism meets the processing requirements of the input stream data to determine whether the current operator parallelism needs to be adjusted.

[0084] S303: Obtain the operation status of operator resources, and determine whether the current operator parallelism meets the processing requirements of input stream data based on the operation status of operator resources, so as to determine whether the current operator parallelism needs to be adjusted.

[0085] In a streaming system, operators rely on underlying physical resources such as the central processing unit (CPU), memory, and processes to complete data processing. If operator resources are insufficient, data processing delays or even task failures may occur. Therefore, real-time monitoring of physical resource usage in Flink jobs is critical to optimizing parallelism and improving job processing performance.

[0086] The operation status of operator resources refers to the state of the current resource usage of the operator, including CPU occupancy, memory usage, process status, etc. Monitoring the operation status of operator resources is one of the important components of the adaptive adjustment of Flink jobs in the embodiment of this application. By monitoring indicators such as CPU, memory, and threads, it is determined whether the current resources meet the processing requirements of the input stream data, and key data support is provided for the dynamic adjustment of the operator parallelism. For example, if the CPU occupancy rate is higher than the first preset threshold, it means that the operator task processing pressure is too large and the current operator parallelism needs to be adjusted; if the memory usage rate is higher than the second preset threshold, it means that the current memory pressure is high, the operator may need more resources, and the current operator parallelism needs to be adjusted; if the number of process threads is much higher than the third preset threshold of the reasonable system load, it means that the resource configuration is unreasonable and the current operator parallelism needs to be adjusted.

[0087] By obtaining the resource operation status of the operator in real time and combining it with the processing requirements of the input stream data, it is possible to intelligently determine whether the current parallelism of the operator is appropriate. This dynamic adjustment mechanism can effectively respond to traffic changes, ensure the efficient operation of the system and the rational use of resources, and improve the flexibility and processing performance of the entire Flink job.

[0088] S304: Obtain data consumption delay, and determine whether the current operator parallelism meets the processing requirements of the input stream data based on the data consumption delay, so as to determine whether the current operator parallelism needs to be adjusted.

[0089] Since data processing in Flink jobs is completed by multiple operators in collaboration, the processing speed of each operator may be different, which may cause speed mismatch between upstream and downstream operators. For example, if the processing speed of the downstream operator is slower than the output speed of the upstream operator, data will accumulate and delay consumption. Therefore, the embodiment of the present application monitors the data consumption delay, promptly discovers problems, and adjusts the operator parallelism in time to solve the problem of consumption delay. Specifically, by setting a delay threshold, when it is monitored that the consumption delay of a certain operator exceeds the delay threshold, it can be considered that the current processing capacity of the operator does not meet the demand. If the operator is higher than the delay threshold for a period of time, it is determined that the current operator parallelism needs to be adjusted to improve the processing capacity.

[0090] S305: Based on the weights corresponding to the factors that need to be adjusted for the current operator parallelism, a settlement result is calculated, where the factors include: adaptation relationship, operator resource operation status, and data consumption delay status.

[0091] In the embodiment of the present application, different weights are pre-assigned to each factor for calculation to comprehensively measure whether the operator parallelism needs to be adjusted. Specifically, the first mark is used to identify the result determined based on the adaptation relationship, the second mark is used to identify the result determined based on the operation of the operator resources, and the third mark is used to identify the result determined based on the data consumption delay cleanup. The values ​​of the first mark, the second mark, and the third mark are 1 or 0, 1 indicates that the current operator parallelism needs to be adjusted, and 0 indicates that the current operator parallelism does not need to be adjusted.

[0092] S306: In response to the calculation result exceeding the preset value, determine whether the current operator parallelism needs to be adjusted, and determine the target operator parallelism based on the data processing volume and the total amount of data of each operator in the Flink job.

[0093] Assume that the first mark is given a weight of 20%, the second mark is given a weight of 40%, and the third mark is given a weight of 40%. Then determine whether the calculation result is close to 1 or 0. If it is close to 1, it needs to be adjusted, and if it is close to 0, it does not need to be adjusted. That is, the preset value is set to 0.5. If it is greater than 0.5, it is close to 1, and if it is less than 0.5, it is close to 0. On this basis, when the first mark is 1, the second mark is 0, and the third mark is 1, after calculation, the calculation result is 1*20%+0+40%+1*40%=0.6. The calculation result is greater than 0.5, and it is determined that the current operator parallelism needs to be adjusted.

[0094] S307: Adjust the current operator parallelism to the target operator parallelism, and perform parallel processing of input stream data based on the target operator parallelism.

[0095] exist Figure 3In the process shown, whether the operator parallelism needs to be adjusted is not only determined based on the adaptation relationship between the total amount of data and the operator parallelism, but also the operation status of the operator resources and the data consumption delay are comprehensively considered. The multi-dimensional judgment basis makes the adjustment decision more accurate and can better balance the job performance. Specifically, by assigning corresponding weights to each judgment factor and calculating the settlement result, this weight calculation method can comprehensively consider the impact of different factors on system performance, so that the adjusted operator parallelism is more in line with actual needs. It can be seen that the above technical solution realizes the dynamic adjustment and optimization of operator parallelism through technical means such as real-time monitoring of input stream data, multi-dimensional judgment basis, weight calculation and precise adjustment, and automatic adjustment and parallel processing, thereby improving the data processing performance of Flink jobs.

[0096] See also Figure 4 As shown, Figure 4 The overall flow chart of a Flink job provided in an embodiment of the present application specifically includes the following steps:

[0097] Step 1: Data flows into the Fink job.

[0098] Step 2: Preprocess the input data stream and calculate the data weight of each data type within the time window.

[0099] Step 3: Data monitoring.

[0100] Step 3.1: Calculate the total amount of input stream data S and the data proportion X of each data type within the time window.

[0101] Step 3.2: Read the preset carrying capacity M and the preset data type planned distribution ratio PX.

[0102] Step 3.3: Determine whether the total amount of data S is greater than the preset carrying capacity M. If so, record the data amount exceeding limit flag flag1.

[0103] Step 3.4: Determine whether the data ratio X of each data type is greater than the corresponding preset data type planned distribution ratio PX, and if so, record the data ratio over-limit flag flag2.

[0104] Step 3.5: Enter the data adaptation discriminator. According to the status of flag1 and flag2, the preset judgment rules are used to evaluate the difference between the data ratio X of each data type and the preset data type planned distribution ratio PX. Combined with the size of the total data S, evaluate whether the parallelism needs to be adjusted.

[0105] Step 3.6: If it is suitable, no adjustment is required and jump directly to step 4.

[0106] Step 3.7: Adjust the type field of the target data type to divide the target data type into at least two sub-data types. Conversely, when the data corresponding to the target data type meets expectations, change the at least two sub-data types back to the target data type.

[0107] Step 3.8: Operators are calculated in parallel according to the grouping type. As the data type increases, the data distribution becomes more even, the proportions meet expectations, and the data skew problem disappears.

[0108] Step 3.9: According to the adaptation relationship between the total amount of data S and the parallelism, determine whether the current operator parallelism needs to be adjusted. If so, record the parallelism adjustment flag flag3.

[0109] Step 4: Get the preprocessed input stream data.

[0110] Step 5: Operator monitoring.

[0111] Step 5.1: Listen to adjust the parallelism flag flag3.

[0112] Step 5.2: Monitor the operation status of operator resources and record it as flag4, such as monitoring the CPU, memory, and process of the Flink physical layer machine.

[0113] Step 5.3: Monitor data consumption delay and record it as flag5 to monitor data source data consumption delay. If data consumption delay occurs, it means that the running speeds of upstream and downstream operators do not match, and the current operator parallelism needs to be adjusted.

[0114] Step 5.4: Enter the parallelism adaptation discriminator, match the preset weight judgment function according to the status of flag3, flag4 and flag5, and determine whether the current operator parallelism needs to be adjusted.

[0115] Step 5.5: Calculate the target operator parallelism.

[0116] Step 5.6: Adjust the current operator parallelism for elastic scaling.

[0117] Step 5.7: Performance adaptation.

[0118] Step 6: Execute the operator calculation logic.

[0119] Step 7: The Flink job outputs data.

[0120] See also Figure 5 As shown, an embodiment of the present application provides an adaptive adjustment device for a Flink job, the device comprising:

[0121] The data statistics module 501 is used for counting the total number of input stream data in the time window in response to the input data stream entering the time window;

[0122] The condition determination module 502 is used to determine whether the current operator parallelism meets the processing requirements of the input stream data based on the adaptation relationship between the total amount of data and the operator parallelism in response to the total amount of data being greater than the preset carrying capacity, so as to determine whether the current operator parallelism needs to be adjusted;

[0123] The parallelism determination module 503 is used to determine the target operator parallelism based on the data processing amount and the total amount of data of each operator in the Flink job in response to the need to adjust the current operator parallelism;

[0124] The parallelism adjustment module 504 is used to adjust the current operator parallelism to the target operator parallelism, and perform parallel processing of input stream data based on the target operator parallelism.

[0125] In some implementations of the embodiments of the present application, the method further includes:

[0126] In response to the input data stream entering the time window, the data ratio and data proportion weight of each data type in the input stream data within the time window are counted;

[0127] In response to the fact that a data ratio of at least one data type in the input stream data exceeds a corresponding preset planned distribution ratio of the data type, the data type exceeding the preset planned distribution ratio of the data type is used as a target data type;

[0128] In response to the data proportion weight of the target data type exceeding the corresponding preset value, the target data type is divided into at least two sub-data types, the data corresponding to the at least two sub-data types in the input stream data constitute the data corresponding to the target data type in the input stream data, and the data corresponding to the at least two sub-data types in the input stream data have no intersection.

[0129] In some implementations of the embodiments of the present application, the target data type is divided into at least two sub-data types, including:

[0130] The target data type is divided into at least two sub-data types by adding different fields to the type field of the target data type.

[0131] In some implementations of the embodiments of the present application, the method further includes:

[0132] In response to the fact that the sum of the data corresponding to the at least two sub-data types in the input stream data does not exceed the preset data type planned distribution ratio corresponding to the target data type, the at least two sub-data types are merged into the target data type.

[0133] In some implementations of the embodiments of the present application, the method further includes:

[0134] Obtain the operation status of operator resources, and based on the operation status of operator resources, determine whether the current operator parallelism meets the processing requirements of input stream data to determine whether the current operator parallelism needs to be adjusted.

[0135] In some implementations of the embodiments of the present application, the method further includes:

[0136] Obtain the data consumption delay, and based on the data consumption delay, determine whether the current operator parallelism meets the processing requirements of the input stream data to determine whether the current operator parallelism needs to be adjusted.

[0137] In some implementations of the embodiments of the present application, in response to the current operator parallelism needing to be adjusted, the following steps are performed:

[0138] The settlement result is calculated based on the weights corresponding to the factors that need to be adjusted for the current operator parallelism. The factors include: adaptation relationship, operator resource operation status, and data consumption delay.

[0139] In response to the calculation result exceeding the preset value, it is determined whether the current operator parallelism needs to be adjusted.

[0140] like Figure 6 As shown, an embodiment of the present application also provides an adaptive adjustment device for Flink jobs, including: a memory 601 and a processor 602; wherein the memory 601 is used to store programs; the processor 602 is used to execute the programs in the memory to implement the various steps of the adaptive adjustment method for Flink jobs provided in the embodiment of the present application.

[0141] The embodiment of the present application further provides a readable storage medium on which a computer program is stored. When the computer program is executed by a processor, each step of the adaptive adjustment method of a Flink job provided in the embodiment of the present application is implemented.

[0142] Finally, it should be noted that, in the embodiments of the present application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements that are not clearly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements limited by the statement "comprise one..." do not exclude the presence of other identical elements in the process, method, article or equipment including the elements.

[0143] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for adaptively adjusting a Flink job, characterized in that: The method comprises: In response to an input data stream entering a time window, counting the total number of data of the input stream data within the time window; In response to the total amount of data being greater than the preset carrying capacity, judging whether the current operator parallelism meets the processing requirements of the input stream data based on the adaptation relationship between the total amount of data and the operator parallelism, so as to determine whether the current operator parallelism needs to be adjusted; In response to the current operator parallelism needing to be adjusted, a target operator parallelism is determined based on the data processing amount of each operator in the Flink job and the total amount of data; The current operator parallelism is adjusted to the target operator parallelism, and the input stream data is processed in parallel based on the target operator parallelism.

2. The method according to claim 1, characterized in that The method further comprises: In response to the input data stream entering the time window, counting the data proportion and data proportion weight of each data type in the input stream data within the time window; In response to the fact that a data ratio of at least one data type in the input stream data exceeds a corresponding preset data type planned distribution ratio, the data type exceeding the preset data type planned distribution ratio is used as a target data type; In response to the data proportion weight of the target data type exceeding the corresponding preset value, the target data type is divided into at least two sub-data types, the data corresponding to the at least two sub-data types in the input stream data constitute the data corresponding to the target data type in the input stream data, and the data corresponding to the at least two sub-data types in the input stream data have no intersection.

3. The method according to claim 2, characterized in that The dividing the target data type into at least two sub-data types comprises: The target data type is divided into the at least two sub-data types by adding different fields in the type field of the target data type.

4. The method according to claim 2, characterized in that: The method further comprises: In response to the fact that the ratio of the data corresponding to the at least two sub-data types and the data in the input stream data does not exceed the preset data type planned distribution ratio corresponding to the target data type, the at least two sub-data types are merged into the target data type.

5. The method according to claim 1, characterized in that: The method further comprises: The operation status of the operator resources is obtained, and based on the operation status of the operator resources, it is determined whether the current operator parallelism meets the processing requirements of the input stream data, so as to determine whether the current operator parallelism needs to be adjusted.

6. The method according to claim 1, characterized in that: The method further comprises: The data consumption delay is obtained, and based on the data consumption delay, it is determined whether the current operator parallelism meets the processing requirements of the input stream data, so as to determine whether the current operator parallelism needs to be adjusted.

7. The method according to claims 5 and 6, characterized in that The step of responding that the current operator parallelism needs to be adjusted includes: Based on the weights corresponding to the factors that need to be adjusted in determining the current operator parallelism, a settlement result is calculated, wherein the factors include: the adaptation relationship, the operation status of the operator resources, and the data consumption delay; In response to the calculation result exceeding a preset value, it is determined whether the current operator parallelism needs to be adjusted.

8. An adaptive adjustment device for Flink jobs, characterized in that: The device comprises: A data statistics module, configured to count the total number of data of the input stream data within the time window in response to the input data stream entering the time window; A condition determination module, configured to determine, in response to the total amount of data being greater than a preset carrying capacity, whether the current operator parallelism meets the processing requirements of the input stream data based on the adaptation relationship between the total amount of data and the operator parallelism, so as to determine whether the current operator parallelism needs to be adjusted; A parallelism determination module is used to determine a target operator parallelism based on the data processing amount of each operator in the Flink job and the total amount of data in response to the need to adjust the current operator parallelism; The parallelism adjustment module is used to adjust the current operator parallelism to the target operator parallelism, and perform parallel processing of the input stream data based on the target operator parallelism.

9. An adaptive adjustment device for Flink jobs, characterized in that: The device includes a memory and a processor, and the processor is used to execute a program stored in the memory to run the adaptive adjustment method of a Flink job according to any one of claims 1 to 7.

10. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the method for adaptively adjusting a Flink job according to any one of claims 1 to 7 is implemented.

Images