Flink parameter optimization method, system, device and storage medium based on YARN scheduling

Abstract

The present application relates to a flink parameter optimization method, system, device and storage medium based on yarn scheduling, the method comprising: obtaining a task request, starting a flink application through a yarn cluster according to the task request; obtaining the indicator parameters during the operation of the flink application in real time; and dynamically optimizing the initial parameters of the flink application according to the corresponding preset adjustment strategy based on whether the indicator parameters can meet the specification requirements. Through this application, the problem of being unable to dynamically optimize parameters according to the operating conditions is solved, and the effect of being able to dynamically optimize parameters according to the actual operating conditions during the operation of the flink application is achieved.

Description

Technical Field

[0001] The present application relates to the field of stream computing technology, and in particular to a Flink parameter optimization method, system, device and storage medium based on YARN scheduling. Background Art

[0002] Flink is a distributed processing engine that executes arbitrary streaming data programs in a data-parallel and pipelined manner. It is used to perform stateful computations on unbounded and bounded data streams, and therefore has many applications in real-time statistics, analysis, or recommendation. At the same time, in order to fully utilize application resources in actual applications, it is usually necessary to optimize the parameter configuration of the Flink application.

[0003] In the prior art, the parameters of the Flink application are usually set manually according to a fixed configuration strategy to achieve static resource configuration. Therefore, during the operation of the Flink application, it is impossible to adjust the parameters for optimization according to the actual operation situation.

[0004] With regard to the problem in related technologies that parameters cannot be dynamically optimized according to operating conditions, no effective solution has been proposed so far. Summary of the invention

[0005] In this embodiment, a Flink parameter optimization method, system, device and storage medium based on YARN scheduling are provided to solve the problems in the related art.

[0006] In the first aspect, a Flink parameter optimization method based on YARN scheduling is provided in this embodiment, which is characterized by comprising:

[0007] Get the task request, and start the Flink application through the YARN cluster according to the task request;

[0008] Obtain the indicator parameters of the flink application in real time during its operation;

[0009] Based on whether the indicator parameters meet the specification requirements, the initial parameters of the Flink application are dynamically optimized according to the corresponding preset adjustment strategy.

[0010] In some of the embodiments, the initial parameters include the number of TaskManagers, the number of slots on each TaskManager, the number of CPU cores used by each TaskManager, and the memory used by each TaskManager.

[0011] In some of the embodiments, the dynamically optimizing the initial parameters of the Flink application according to the corresponding preset adjustment strategy based on whether the indicator parameters can meet the specification requirements includes:

[0012] When the indicator parameter does not meet the specification requirement, adjusting the initial parameter according to a corresponding preset adjustment strategy to obtain a first optimized parameter;

[0013] If the first optimization parameter meets the specification requirement, further optimizing the first optimization parameter to obtain a second optimization parameter;

[0014] If the first optimization parameter does not meet the specification requirements, an abnormal alarm is issued.

[0015] In some embodiments, the adjusting the initial parameters to obtain the first optimized parameters includes:

[0016] Double the number of TaskManagers in the initial parameters, while keeping the number of slots on each TaskManager and the number of CPU cores used by each TaskManager unchanged;

[0017] When the throughput in the indicator parameter reaches the specification requirement, obtaining the first optimization parameter;

[0018] When the throughput in the indicator parameter does not meet the specification requirements and the number of TaskManagers reaches the upper limit, the number of slots on each TaskManager and the number of CPU cores used by each TaskManager are doubled until the throughput meets the specification requirements, thereby obtaining the first optimization parameter.

[0019] In some embodiments, further optimizing the first optimization parameter to obtain a second optimization parameter includes:

[0020] By using the binary method, the number of TaskManagers in the first optimization parameter is halved, and the number of slots on each TaskManager and the number of CPU cores used by each TaskManager in the first optimization parameter are doubled until the number of CPU cores exceeds the upper limit;

[0021] When the number of CPU cores exceeds the upper limit and the throughput in the indicator parameter reaches the specification requirement, the YARN cluster cannot provide more application resources, and the first optimization parameter before exceeding the upper limit is used as the second optimization parameter.

[0022] In some embodiments, the method further comprises:

[0023] When the business requirements can be met, the number of CPU cores used by each TaskManager in the second optimization parameter is halved through the binary search method, and then the memory used by each TaskManager in the second optimization parameter is adaptively adjusted to achieve the use of the least memory to meet the business requirements, and obtain the third optimization parameter.

[0024] In some embodiments, the method further comprises:

[0025] When the indicator parameters meet the specification requirements, the initial parameters are optimized according to the corresponding preset adjustment strategy to obtain fourth optimized parameters.

[0026] In some embodiments, the optimizing the initial parameters to obtain the fourth optimized parameters includes:

[0027] By using the binary method, the number of TaskManagers in the initial parameters is halved, and at the same time, the number of slots on each TaskManager and the number of CPU cores used by each TaskManager in the initial parameters are doubled until the number of CPU cores exceeds the upper limit;

[0028] When the number of CPU cores exceeds the upper limit and the throughput in the indicator parameter reaches the specification requirement, the YARN cluster cannot provide more application resources, and the initial parameter before exceeding the upper limit is used as the fourth optimization parameter.

[0029] In some embodiments, the method further comprises:

[0030] When the business requirements can be met, the number of CPU cores used by each TaskManager in the fourth optimization parameter is halved through the dichotomy method, and the memory used by each TaskManager in the fourth optimization parameter is adaptively adjusted to achieve the goal of using the least memory to meet the business requirements, thereby obtaining the fifth optimization parameter.

[0031] In some embodiments, the method further comprises:

[0032] Each time the number of TaskManagers is adjusted, the memory usage in the indicator parameter is compared with a preset value;

[0033] If the memory usage exceeds the preset value, the memory used by each TaskManager is doubled;

[0034] If the memory usage does not exceed the preset value, the memory used by each TaskManager is kept unchanged.

[0035] In some embodiments, the method further comprises:

[0036] Based on the real data collected from actual application scenarios, the training sample set is obtained regularly;

[0037] The training sample set is configured accordingly by the method, and target recognition is performed based on the training sample set.

[0038] Second, in this embodiment, a flink parameter optimization system based on yarn scheduling is provided, including:

[0039] The resource application module is used to obtain a task request and start the Flink application through the YARN cluster according to the task request;

[0040] The parameter collection module is used to obtain the indicator parameters of the flink application in real time during its operation;

[0041] The dynamic optimization module is used to dynamically optimize the initial parameters of the flink application according to the corresponding preset adjustment strategy based on whether the indicator parameters meet the specification requirements.

[0042] In a third aspect, a computer device is provided in this embodiment, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the Flink parameter optimization method based on YARN scheduling described in the first aspect is implemented.

[0043] In a fourth aspect, a storage medium is provided in this embodiment, on which a computer program is stored. When the program is executed by a processor, the Flink parameter optimization method based on YARN scheduling described in the first aspect is implemented.

[0044] Compared with the related art, the flink parameter optimization method, system, device and storage medium based on yarn scheduling provided in this embodiment include obtaining a task request, starting the flink application through the yarn cluster according to the task request; obtaining the index parameters of the flink application in real time during operation; and dynamically optimizing the initial parameters of the flink application according to the corresponding preset adjustment strategy based on whether the index parameters can meet the specification requirements. By deploying the flink application on the yarn cluster, dynamically optimizing the initial parameters of the flink application according to the corresponding preset adjustment strategy, and scheduling the flink application resources by the yarn cluster, the problem of being unable to dynamically optimize parameters according to the operating conditions is solved, and the effect of being able to dynamically optimize the operating parameters according to the actual operating conditions is achieved.

[0045] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0047] Figure 1 The hardware structure block diagram of a Flink parameter optimization method based on YARN scheduling in one embodiment;

[0048] Figure 2 This is a flowchart of a Flink parameter optimization method based on YARN scheduling in one embodiment;

[0049] Figure 3 A flowchart of a preset adjustment strategy corresponding to an embodiment when an indicator parameter does not meet specification requirements;

[0050] Figure 4 A flowchart of a corresponding preset adjustment strategy when an indicator parameter meets specification requirements in an embodiment;

[0051] Figure 5 It is a schematic diagram of the execution of a Flink parameter optimization method based on YARN scheduling in a preferred embodiment;

[0052] Figure 6 The following is a structural block diagram of a Flink parameter optimization system based on YARN scheduling in an embodiment.

[0053] In the figure: 10, resource application module; 20, parameter collection module; 30, dynamic optimization module. DETAILED DESCRIPTION

[0054] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0055] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the", "these" and the like in this application do not represent quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether directly or indirectly. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. Usually, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0056] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 : is a hardware structure diagram of a terminal of the Flink parameter optimization method based on YARN scheduling in this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) processor 102 and memory 104 for storing data, wherein processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0057] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the Flink parameter optimization method based on yarn scheduling in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0058] The transmission device 106 is used to receive or send data via a network. The above network includes a wireless network provided by the communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0059] Yarn is a distributed resource management and scheduling system that is used to improve resource utilization in a distributed cluster environment. The resources used include memory, IO, network, disk, etc. The granularity of resource scheduling in Yarn is container. Based on the application resource application received, the ResourceManager in Yarn runs the container in the NodeManager and starts the ApplicationManager in the container. The ApplicationManager then performs specific resource configuration for the application.

[0060] In this embodiment, a Flink parameter optimization method based on YARN scheduling is provided. Figure 2 is the corresponding flow chart of this implementation, such as Figure 2 As shown, the process includes the following steps:

[0061] Step S210: Obtain a task request, and start the Flink application through the YARN cluster according to the task request.

[0062] Specifically, the flink client submits a task request to the yarn cluster and applies for application resources. After receiving the task request, the ResourceManager in the yarn cluster starts the ApplicationManager in the allocated container, and then starts the JobManager and TaskManager of the flink application in the ApplicationManager according to the task request and the applied resources. In actual operation, the JobManager assigns tasks to the TaskManager, and each TaskManager executes the task.

[0063] It is important to know that since the Flink application is deployed on the YARN cluster and runs through a virtual machine, when no task resources are requested to start the Flink application, the Flink application will not occupy physical machine resources, so resources can be more fully utilized.

[0064] Step S220, obtaining indicator parameters during the running of the flink application in real time.

[0065] Specifically, the above indicator parameters include throughput and memory usage. Throughput refers to the amount of data successfully transmitted per unit time, and memory usage refers to the memory usage on TaskManager in the above flink application. The obtained real-time indicator parameters are stored in the time series database influxdb.

[0066] Furthermore, according to actual application requirements, indicator parameters can also include GC ratio and checkpoint duration, where GC ratio is related to memory usage and checkpoint duration is related to throughput.

[0067] Step S230: Based on whether the indicator parameters meet the specification requirements, the initial parameters of the Flink application are dynamically optimized according to the corresponding preset adjustment strategy.

[0068] Specifically, the above-mentioned specification requirements refer to the throughput required to be achieved, which is usually pre-set according to user needs. Based on whether the indicator parameters can meet the specification requirements, that is, whether the real-time throughput during the operation of the flink application meets the pre-set throughput, the initial parameters of the flink application are dynamically optimized according to the corresponding different preset adjustment strategies. The above-mentioned preset adjustment strategy specifically includes: first adjusting the initial parameters to meet the specification requirements, and then adjusting the initial parameters of all flink applications through linkage, while still meeting the specification requirements, gradually improving the utilization rate of flink application resources and reducing memory resource consumption. The above-mentioned initial parameters refer to the initial running parameters when the yarn cluster starts the flink application for the first time according to the task request.

[0069] Through the above steps, this embodiment obtains a task request, starts a Flink application through a YARN cluster according to the task request, obtains indicator parameters during the running of the Flink application in real time, and dynamically optimizes the initial parameters of the Flink application according to corresponding preset adjustment strategies based on whether the indicator parameters can meet the specification requirements.

[0070] The existing flink application resource configuration is usually to set a fixed configuration strategy by manually estimating the resources that may be needed before the flink application runs, so the flink application cannot adjust parameters accordingly according to different application environments or actual running conditions during operation. This embodiment provides an effective supplement based on the existing technology, by obtaining the index parameters of the flink application in real time during operation, and based on the real-time index parameters, preset corresponding different adjustment strategies to dynamically optimize the initial parameters, thus solving the problem in the existing technology that the parameters cannot be dynamically optimized according to the running conditions.

[0071] In some of the embodiments, the initial parameters include the number of TaskManagers, the number of slots on each TaskManager, the number of CPU cores used by each TaskManager, and the memory used by each TaskManager.

[0072] Specifically, the initial parameters are set to include four TaskManagers, each TaskManager is assigned a slot, each TaskManager uses a CPU core, and each TaskManager uses 2G memory. The yarn cluster first configures resources for the flink application according to the above initial parameter settings in the task request and starts the flink application. In other embodiments, the initial parameters can also be configured accordingly according to the application environment, which is not limited.

[0073] Each TaskManager in the flink application is a JVM process, and one or more threads can be executed in it. Slot is the number of slots. Specifically, the number of slots is controlled to control how many threads a TaskManager can receive. During operation, the flink application will allocate management memory to each slot, and adjust the above initial parameters in a linked manner to meet the specification requirements while using the least memory, so that the flink application can run with fewer processes (number of TaskManagers) and more threads (number of slots) to achieve reasonable and optimized resource allocation and performance optimization.

[0074] Furthermore, since the Flink application is deployed on the YARN cluster and runs through a virtual machine, in the YARN scheduling environment, each TaskManager runs in a container, so the number of CPU cores used by each TaskManager can be configured through the yarn.containers.vcores instruction.

[0075] In some of the embodiments, based on whether the indicator parameters meet the specification requirements, the initial parameters of the Flink application are dynamically optimized according to the corresponding preset adjustment strategy. When the indicator parameters do not meet the specification requirements, the corresponding preset adjustment strategy is as follows: Figure 3 As shown, the following steps are included:

[0076] Step S310: When the indicator parameters do not meet the specification requirements, the initial parameters are adjusted according to the corresponding preset adjustment strategy.

[0077] Specifically, when the throughput in the indicator parameter does not reach the throughput required by the specification, the number of TaskManagers in the initial parameters is doubled, while the number of slots on each TaskManager and the number of CPU cores used by each TaskManager in the initial parameters remain unchanged.

[0078] It is important to know that in the YARN scheduling environment, each time the operating parameters are adjusted, the Flink application must resubmit the task request and apply for resources. The YARN cluster will release the previously applied application each time and re-allocate resources such as the number of TaskManagers to the Flink application.

[0079] Furthermore, each time the number of TaskManagers is adjusted, the memory usage in the indicator parameter is compared with the preset value;

[0080] If the memory usage exceeds the preset value, the memory used by each TaskManager will be doubled;

[0081] If the memory usage does not exceed the preset value, the memory used by each TaskManager remains unchanged.

[0082] Specifically, the preset memory usage is 70%. Each time the number of TaskManagers is adjusted, the memory usage on each TaskManager will also change. If the memory usage exceeds 70% at this time, the memory used by each TaskManager is doubled. If it does not exceed 70%, the memory remains unchanged. In other embodiments, the memory usage can be other values, such as: 60%, 80%, 90%, etc., and there is no limitation on this.

[0083] Furthermore, the method further comprises the following steps:

[0084] Step S311, when the throughput in the indicator parameter reaches the specification requirement, a first optimization parameter is obtained.

[0085] Specifically, if the throughput can reach the specification requirement after adjusting the number of TaskManagers in the above step S310, the adjusted initial parameter is used as the first optimization parameter.

[0086] Step S312: When the throughput in the indicator parameter does not meet the specification requirements and the number of TaskManagers reaches the upper limit, the number of slots on each TaskManager and the number of CPU cores used by each TaskManager are doubled until the throughput meets the specification requirements, thereby obtaining the first optimization parameters.

[0087] Specifically, if the specification requirements cannot be met by simply adjusting the number of TaskManagers in the above step S310, and the number of TaskManagers has reached the resource limit of the yarn cluster, then further double the number of slots on each TaskManager and the number of CPU cores used by each TaskManager until the throughput meets the specification requirements. The initial parameters are adjusted through the above steps to increase the throughput at runtime. Assuming that the throughput set by the user in the specification requirements is 3000 bit / s, then until the throughput at runtime reaches 3000 bit / s, it means that the throughput meets the specification requirements, and the first optimization parameter is obtained. In fact, the throughput in the specification requirements can be set independently according to user requirements.

[0088] Among them, YARN cluster resources include CPU core count, memory and other resources, which are limited by the conditions of physical machines deployed at the application site.

[0089] Step S320: If the first optimization parameter meets the specification requirements, the first optimization parameter is further optimized to obtain a second optimization parameter.

[0090] Specifically, after the above steps S310 and S312, if the throughput currently achieved by running the first optimization parameter can reach the throughput set by the user in the specification requirements, it means that the first optimization parameter meets the specification requirements, then further optimizing the first optimization parameter to obtain the second optimization parameter includes the following steps:

[0091] Use the binary method to halve the number of TaskManagers in the first optimization parameter, and at the same time double the number of slots on each TaskManager and the number of CPU cores used by each TaskManager in the first optimization parameter until the number of CPU cores exceeds the upper limit;

[0092] When the number of CPU cores exceeds the upper limit and the throughput in the indicator parameter reaches the specification requirement, the YARN cluster cannot provide more application resources and uses the first optimization parameter before exceeding the upper limit as the second optimization parameter.

[0093] Specifically, the above steps reduce the number of TaskManagers so that more slot threads can run simultaneously on one TaskManager, thereby improving the resource utilization of each TaskManager. Furthermore, each time the number of TaskManagers is adjusted, the memory usage in the indicator parameter needs to be compared with the preset value, and then the memory used by each TaskManager is doubled or kept unchanged.

[0094] Step S330: If the first optimization parameter does not meet the specification requirements, an abnormal alarm is issued.

[0095] Specifically, if the first optimization parameter still does not meet the specification requirements after the parameter adjustment in the above steps, it means that the current YARN cluster resources are insufficient, and the resources of the entire YARN cluster are insufficient to make the actual running throughput reach the throughput set by the user in the specification requirements, and the Flink application can no longer apply for resources, then the alarm throws an exception.

[0096] Furthermore, the above strategy also includes the following steps:

[0097] Step S340, when the business requirements can be met, the number of CPU cores used by each TaskManager in the second optimization parameter is halved by binary search, and the memory used by each TaskManager in the second optimization parameter is adaptively adjusted to achieve the use of the least memory to meet the business requirements, and obtain the third optimization parameter.

[0098] Specifically, after the above steps S310 to S330, the second optimization parameter has met the specification requirements, so each TaskManager can further meet the business requirements of the actual application while using the least number of CPU cores. The business requirements are specifically limited by the actual application scenario. For example, in the application scenario of target recognition, the business values ​​such as recognition time, recognition rate and recognition accuracy are required to be no less than the preset lower limit. If the above business values ​​can reach at least the preset lower limit when running the third optimization parameter in the actual application scenario, it means that the business requirements can be met. At the same time, according to the memory usage required by the user, the memory used by each TaskManager is increased or reduced, and the resource utilization of the flink application is improved by running as many slots (threads) as possible on each TaskManager (process).

[0099] Through this embodiment, when the real-time indicator parameters of the Flink application running with the initial parameters do not meet the specification requirements, according to the corresponding preset adjustment strategy, the first optimization parameter and the second optimization parameter are obtained based on the specification requirements, and then the third optimization parameter is obtained based on the business requirements, so that the third optimization parameter can dynamically optimize the initial parameters according to the indicator parameters in the actual operation process of the application while meeting the specification requirements and business requirements, and at the same time can maximize the use of YARN cluster resources.

[0100] In some of the embodiments, based on whether the indicator parameters meet the specification requirements, the initial parameters of the Flink application are dynamically optimized according to the corresponding preset adjustment strategy. When the indicator parameters meet the specification requirements, Figure 4 As shown, the corresponding preset adjustment strategy includes the following steps:

[0101] Step S410: When the indicator parameters meet the specification requirements, the initial parameters are optimized according to the corresponding preset adjustment strategy to obtain fourth optimized parameters.

[0102] Specifically, the number of TaskManagers in the initial parameters is halved through the binary method, and the number of slots on each TaskManager and the number of CPU cores used by each TaskManager in the initial parameters are doubled until the number of CPU cores exceeds the upper limit;

[0103] When the number of CPU cores exceeds the upper limit and the throughput in the indicator parameter reaches the specification requirement, the yarn cluster cannot provide more application resources and uses the initial parameters before exceeding the upper limit as the fourth optimization parameter. Among them, the throughput in the indicator parameter reaches the specification requirement means that when the Flink application is run with the optimized parameters, the actual running throughput can reach the throughput set by the user in the specification requirement.

[0104] Furthermore, each time the number of TaskManagers is adjusted, the memory usage in the indicator parameter needs to be compared with the preset value, and then the memory used by each TaskManager is doubled or kept unchanged.

[0105] Step S420, when the business requirements can be met, the number of CPU cores used by each TaskManager in the fourth optimization parameter is halved by binary search, and the memory used by each TaskManager in the fourth optimization parameter is adaptively adjusted to achieve the minimum memory used to meet the business requirements, thereby obtaining the fifth optimization parameter.

[0106] Specifically, the business requirements are specifically defined by the actual application scenarios, which also include the business values ​​of specific businesses, and have lower or upper limits on the business values. If, in the actual application scenario, when running the fifth optimization parameter, the above-mentioned business value can at least reach the preset lower or upper limit, it means that the business requirements can be met.

[0107] Through this embodiment, when the real-time indicator parameters of the Flink application running with the initial parameters meet the specification requirements, according to the corresponding preset adjustment strategy, the fourth optimization parameter is obtained based on the specification requirements, and then the fifth optimization parameter is obtained based on the business requirements, so that the fifth optimization parameter can dynamically optimize the initial parameters according to the indicator parameters in the actual operation process of the application while meeting the specification requirements and business requirements, and can maximize the use of YARN cluster resources.

[0108] In some embodiments, the method further comprises the following steps:

[0109] Step S510, regularly acquiring a training sample set based on real data collected in an actual application scenario;

[0110] Specifically, an upper limit of the training sample set is also set to prevent the training samples from occupying too many resources. After the sample threshold is reached, the training sample set is updated in a cyclic covering manner, so that the obtained training sample set can be updated in real time and can better simulate the application environment.

[0111] Furthermore, according to different data sources of the sample sets, including data sources such as kafka, rabbitmq and Hbase, the sample sets of the same data source are stored in the same data source module.

[0112] Step S520: configure the training sample set accordingly through the method, and perform target recognition based on the training sample set.

[0113] Specifically, sample sets are first distinguished according to different field application environments, such as vehicle data samples, face data samples, and MAC (Media Access Control) data samples, and the above method is used to configure the training sample sets accordingly, and based on the training sample sets, the operating parameters in the target recognition task are optimized in the field application environment.

[0114] Furthermore, when the training samples are not fully collected, real data is used for target recognition first, and then the collection of training samples is completed.

[0115] Through this embodiment, it is possible to obtain training sample sets at a regular interval based on real data collected from actual application scenarios, and when the above method switches tasks between different application scenarios, it only needs to perform corresponding configuration based on the training sample sets, and the above method can be used to perform operating parameter optimization in target recognition tasks in different application scenarios.

[0116] The present embodiment is described and illustrated below through preferred embodiments.

[0117] This preferred embodiment provides a flink parameter optimization method based on yarn scheduling. Figure 5 is a schematic diagram of the corresponding method execution, the method comprising the following steps:

[0118] Step S610: Obtain a task request, and start the Flink application through the YARN cluster according to the task request.

[0119] Step S620: acquiring a training sample set at regular intervals based on real data collected from actual application scenarios.

[0120] Step S630: configure the training sample set accordingly using the method.

[0121] Step S640: Obtain the indicator parameters during the running of the Flink application in real time and store them in the time series database influxdb.

[0122] Step S650: Based on whether the indicator parameters meet the specification requirements, the initial parameters of the Flink application are dynamically optimized according to the corresponding preset adjustment strategy.

[0123] Specifically, the corresponding preset adjustment strategies are described as examples in all the above embodiments.

[0124] Step S660: Send the corresponding optimization parameters obtained through dynamic optimization to the Flink client, and the Flink client resubmits the task request according to the optimization parameters.

[0125] It should be noted that the steps shown in the above process or the flowchart in the attached figure can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here. For example, the order of step S610 and step S620 can be interchanged, first obtaining the training sample set regularly and then starting the Flink application through the yarn cluster according to the task request.

[0126] In this embodiment, a flink parameter optimization system based on yarn scheduling is also provided, which is used to implement the above embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. The terms "module", "unit", "subunit", etc. used below can implement a combination of software and / or hardware for predetermined functions. Although the systems described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0127] Figure 6 is a structural block diagram of the Flink parameter optimization system based on YARN scheduling in this embodiment, such as Figure 6 As shown, the system includes a resource application module 10, a parameter collection module 20 and a dynamic optimization module 30, wherein:

[0128] Resource application module 10, used to obtain task requests and start Flink applications through the YARN cluster according to the task requests;

[0129] The parameter collection module 20 is used to obtain the indicator parameters of the flink application in real time during operation;

[0130] The dynamic optimization module 30 is used to dynamically optimize the initial parameters of the Flink application according to the corresponding preset adjustment strategy based on whether the indicator parameters meet the specification requirements.

[0131] Through the system provided in this embodiment, the Flink application is started through the YARN cluster according to the task request, the indicator parameters of the Flink application during operation are obtained in real time, and based on the real-time indicator parameters, corresponding different adjustment strategies are preset to dynamically optimize the initial parameters, thereby solving the problem in the prior art that parameters cannot be dynamically optimized according to the operating conditions.

[0132] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0133] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0134] In addition, in combination with the Flink parameter optimization method based on YARN scheduling provided in the above embodiment, a storage medium can also be provided in this embodiment to implement. The storage medium stores a computer program; when the computer program is executed by the processor, any one of the Flink parameter optimization methods based on YARN scheduling in the above embodiment is implemented.

[0135] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0136] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

[0137] The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various locations in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

[0138] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A flink parameter optimization method based on yarn scheduling, characterized in that: include: Get the task request, and start the Flink application through the YARN cluster according to the task request; Obtain the indicator parameters of the flink application in real time during its operation; Based on whether the indicator parameters meet the specification requirements, the initial parameters of the flink application are dynamically optimized according to the corresponding preset adjustment strategy; specifically including: When the indicator parameter does not meet the specification requirements, the initial parameter is adjusted according to the corresponding preset adjustment strategy to obtain the first optimization parameter; wherein, the following steps are included: Double the number of TaskManagers in the initial parameters, while keeping the number of slots on each TaskManager and the number of CPU cores used by each TaskManager unchanged; When the throughput in the indicator parameter reaches the specification requirement, obtaining the first optimization parameter; When the throughput in the indicator parameter does not meet the specification requirement and the number of TaskManagers reaches the upper limit, the number of slots on each TaskManager and the number of CPU cores used by each TaskManager are doubled until the throughput meets the specification requirement, thereby obtaining the first optimization parameter; If the first optimization parameter meets the specification requirement, further optimizing the first optimization parameter to obtain a second optimization parameter; If the first optimization parameter does not meet the specification requirements, an abnormal alarm is issued.

2. According to the flink parameter optimization method based on yarn scheduling according to claim 1, it is characterized in that: The initial parameters include the number of TaskManagers, the number of slots on each TaskManager, the number of CPU cores used by each TaskManager, and the memory used by each TaskManager.

3. The flink parameter optimization method based on yarn scheduling according to claim 1 is characterized in that: The further optimizing the first optimization parameter to obtain a second optimization parameter includes: By using the binary method, the number of TaskManagers in the first optimization parameter is halved, and the number of slots on each TaskManager and the number of CPU cores used by each TaskManager in the first optimization parameter are doubled until the number of CPU cores exceeds the upper limit; When the number of CPU cores exceeds the upper limit and the throughput in the indicator parameter reaches the specification requirement, the YARN cluster cannot provide more application resources, and the first optimization parameter before exceeding the upper limit is used as the second optimization parameter.

4. The flink parameter optimization method based on yarn scheduling according to claim 1 is characterized in that: Also includes: When the business requirements can be met, the number of CPU cores used by each TaskManager in the second optimization parameter is halved through the binary search method, and then the memory used by each TaskManager in the second optimization parameter is adaptively adjusted to achieve the use of the least memory to meet the business requirements, and obtain the third optimization parameter.

5. The flink parameter optimization method based on yarn scheduling according to claim 1 is characterized in that: The dynamically optimizing the initial parameters of the Flink application according to the corresponding preset adjustment strategy based on whether the indicator parameters can meet the specification requirements also includes: When the indicator parameters meet the specification requirements, the initial parameters are optimized according to the corresponding preset adjustment strategy to obtain fourth optimized parameters.

6. The flink parameter optimization method based on yarn scheduling according to claim 5 is characterized in that: The step of optimizing the initial parameters to obtain fourth optimized parameters includes: By using the binary method, the number of TaskManagers in the initial parameters is halved, and at the same time, the number of slots on each TaskManager and the number of CPU cores used by each TaskManager in the initial parameters are doubled until the number of CPU cores exceeds the upper limit; When the number of CPU cores exceeds the upper limit and the throughput in the indicator parameter reaches the specification requirement, the YARN cluster cannot provide more application resources, and the initial parameter before exceeding the upper limit is used as the fourth optimization parameter.

7. The flink parameter optimization method based on yarn scheduling according to claim 5 is characterized in that: Also includes: When the business requirements can be met, the number of CPU cores used by each TaskManager in the fourth optimization parameter is halved through the dichotomy method, and the memory used by each TaskManager in the fourth optimization parameter is adaptively adjusted to achieve the goal of using the least memory to meet the business requirements, thereby obtaining the fifth optimization parameter.

8. The flink parameter optimization method based on yarn scheduling according to any one of claims 1, 3 or 6, characterized in that: Also includes: Each time the number of TaskManagers is adjusted, the memory usage in the indicator parameter is compared with a preset value; If the memory usage exceeds the preset value, the memory used by each TaskManager is doubled; If the memory usage does not exceed the preset value, the memory used by each TaskManager is kept unchanged.

9. The flink parameter optimization method based on yarn scheduling according to claim 1 is characterized in that: Also includes: Based on the real data collected from actual application scenarios, the training sample set is obtained regularly; The training sample set is configured accordingly by the method, and target recognition is performed based on the training sample set.

10. A flink parameter optimization system based on yarn scheduling, characterized in that: include: The resource application module is used to obtain task requests and start the Flink application through the YARN cluster according to the task requests; The parameter collection module is used to obtain the indicator parameters of the flink application in real time during its operation; A dynamic optimization module is used to dynamically optimize the initial parameters of the Flink application according to the corresponding preset adjustment strategy based on whether the indicator parameters meet the specification requirements; specifically, it includes: When the indicator parameter does not meet the specification requirements, the initial parameter is adjusted according to the corresponding preset adjustment strategy to obtain the first optimization parameter; wherein, the following steps are included: Double the number of TaskManagers in the initial parameters, while keeping the number of slots on each TaskManager and the number of CPU cores used by each TaskManager unchanged; When the throughput in the indicator parameter reaches the specification requirement, obtaining the first optimization parameter; When the throughput in the indicator parameter does not meet the specification requirement and the number of TaskManagers reaches the upper limit, the number of slots on each TaskManager and the number of CPU cores used by each TaskManager are doubled until the throughput meets the specification requirement, thereby obtaining the first optimization parameter; If the first optimization parameter meets the specification requirement, further optimizing the first optimization parameter to obtain a second optimization parameter; If the first optimization parameter does not meet the specification requirements, an abnormal alarm is issued.

11. A computer device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the Flink parameter optimization method based on YARN scheduling according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the Flink parameter optimization method based on YARN scheduling according to any one of claims 1 to 9 are implemented.

Images