A resource configuration method and apparatus for a distributed streaming data stream engine

Abstract

This invention provides a resource setting method and apparatus for a distributed streaming data stream engine. The method includes: generating a logical flow graph of the task to be processed based on a received task; obtaining resource information of each flow node in the logical flow graph from the operator resource configuration; merging flow nodes with the same set attributes in the logical flow graph into group nodes to obtain a logical group graph with resource information of each group node; and submitting the logical group graph to a resource scheduler. By setting the operator resource configuration, fine-grained resource settings for the resources required by the task's operators can be easily and quickly implemented. Furthermore, each time the operator resource configuration is updated, the resource information of each flow node in the logical flow graph is updated accordingly. This allows for convenient setting of the resource information of each flow node in the logical flow graph as needed, thus enabling flexible control over the resource information of each flow node and group node, improving the efficiency of resource setting.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a resource setting method, apparatus, computing device and computer-readable storage medium for use in a distributed streaming data stream engine. Background Technology

[0002] As data development capabilities mature, the volume of real-time task development is increasing. Currently, the main open-source computing engines used for real-time data processing include Storm, Spark, and the distributed streaming data engine Flink. Thanks to Flink's maturity in the real-time domain, most real-time platforms are built upon it. In practical applications, the streaming engine client receives tasks to be processed, configures the operators, parallelism, total memory per TaskManager (TM), and the number of slots per TM, and submits the configured tasks to a resource scheduler such as a Yarn cluster. The Yarn cluster then allocates appropriate resources to the tasks based on their configurations.

[0003] Generally, a Yarn cluster distributes the total memory of each TM equally based on the number of slots on each TM. Each slot then executes the corresponding operator in the task according to its allocated resources. However, Yarn cluster resources are limited. Allocating too many resources to an operator for a task will waste resources and crowd out the execution resources of other tasks; conversely, allocating too few resources will result in a slower execution speed for that task.

[0004] Therefore, if the resources of the operators of a task can be configured in a targeted and fine-grained manner, the Yarn cluster can allocate resources of the appropriate size, thereby improving resource utilization.

[0005] In summary, this invention provides a resource setting method for a distributed streaming data stream engine, which allows for simple and quick fine-grained setting of the resources required by the operators of a task, thereby improving the efficiency of resource setting. Summary of the Invention

[0006] This invention provides a resource setting method for a distributed streaming data stream engine, which allows for simple and quick fine-grained setting of the resources required by the operators of a task, thereby improving the efficiency of resource setting.

[0007] In a first aspect, embodiments of the present invention provide a resource setting method applied to a distributed streaming data stream engine, including:

[0008] The stream engine client generates a logical flow graph of the received task to be processed, which includes flow nodes representing operators used to process the task to be processed.

[0009] The stream engine client obtains the resource information of each stream node in the logical flow graph from the operator resource configuration;

[0010] The streaming engine client merges streaming nodes with the same set attributes in the logical streaming graph into group nodes, resulting in a logical group graph with resource information for each group node; wherein, the resource information of the group node is obtained based on the resource information of the streaming nodes in the group node;

[0011] The streaming engine client submits the logical graph to the resource scheduler; the resource scheduler allocates resources for processing the pending tasks according to the logical graph.

[0012] By configuring operator resources, fine-grained resource settings for task operators can be easily and quickly implemented. Furthermore, each time the operator resource configuration is updated, the resource information of each flow node in the logical flow graph is also updated accordingly. This allows for convenient configuration of resource information for each flow node in the logical flow graph as needed. Further, based on the updated logical flow graph, the resource information of each group node in the generated logical group graph is also updated accordingly. The logical group graph configured according to the developer's needs is then submitted to the resource scheduler, which allocates corresponding resources based on the resource settings of each flow node in the logical group graph. In this way, when data traffic or system network speed changes, flexible and fine-grained adjustment of resource information for each flow node and group node can be achieved, improving the efficiency of resource configuration. Subsequently, the Yarn cluster can allocate resources of appropriate sizes based on the resources configured by the developer on the client, thus improving resource utilization.

[0013] Optionally, the method further includes:

[0014] The streaming engine client converts the logical group diagram into a task plan diagram;

[0015] The streaming engine client obtains resource information of each group node from the logical group graph and resource information of each streaming node from the logical flow graph;

[0016] The streaming engine client fills the task schedule graph with the resource information of each group of nodes and the resource information of each streaming node, and then displays the task schedule graph.

[0017] By obtaining the resource information of group nodes and flow nodes from the logical group diagram and logical flow diagram through the flow engine client and populating it into the task schedule diagram, the resource information of group nodes and flow nodes can be displayed in the task schedule diagram, making it convenient for users to view, compare and modify.

[0018] Optionally, the method further includes:

[0019] The streaming engine client receives resource modification information from the user's resource plan display interface;

[0020] The streaming engine client modifies the corresponding operator in the operator resource configuration based on the resource modification information;

[0021] The stream engine client updates the logical graph based on the modified operator resource configuration and sends it to the resource scheduler.

[0022] By receiving user resource modification information from the task scheduler's display interface through the stream engine client, the operator resource configuration can be modified accordingly. This facilitates the subsequent logical flow graph to obtain resource information for each flow node from the operator resource configuration and then update the logical group graph accordingly. Sending the updated logical group graph to the resource scheduler allows the scheduler to allocate resources based on the updated resource information, improving resource utilization.

[0023] Optionally, the streaming engine client generates a logical flow graph of the received tasks based on the tasks to be processed, including:

[0024] The streaming engine client generates the logical flow graph of the task to be processed based on the FlinkJar or FlinkSql package.

[0025] The streaming engine client can generate logical flow graphs of tasks to be processed based on different programming languages, such as FlinkJar and FlinkSql. That is, the resource setting method provided in this embodiment of the invention is applicable to multiple programming languages, thus unifying the resource setting method.

[0026] Optionally, the stream engine client obtains resource information of each group node from the logical group graph and resource information of each stream node from the logical flow graph, including:

[0027] The streaming engine client traverses each group of nodes in the task plan graph, maps each group of nodes in the task plan graph to each group of nodes in the logical group graph, and obtains the resource information of each group of nodes in the logical group graph.

[0028] The streaming engine client traverses each streaming node in the task schedule graph, maps each streaming node in the task schedule graph to a streaming node in the logical flow graph, and obtains the resource information of each streaming node in the logical flow graph. Through this method, the resource information of each group of nodes and each streaming node can be obtained quickly and accurately.

[0029] Optionally, the streaming engine client modifies the corresponding operator in the operator resource configuration based on the resource modification information, including:

[0030] The streaming engine client obtains the resource modification information and the group node corresponding to the resource modification information, distributes the resource modification information equally according to the number of streaming nodes contained in the group node, and updates the resource modification information of the operator corresponding to the streaming node contained in the group node in the operator resource configuration.

[0031] Users can modify the flow nodes simultaneously by modifying the group nodes. This makes modification simple and quick, improving the efficiency of resource configuration.

[0032] Optionally, the streaming engine client modifies the corresponding operator in the operator resource configuration based on the resource modification information, including:

[0033] The streaming engine client obtains the resource modification information and the streaming node corresponding to the resource modification information, and updates the resource modification information to the resource information of the operator corresponding to the streaming node in the operator resource configuration.

[0034] Users can make various modifications to the resources of each flow node, and the modified resource information will be updated accordingly in the operator resource configuration. Modification is simple and quick, improving the efficiency of resource setting.

[0035] Secondly, embodiments of the present invention also provide a resource setting device for a distributed streaming data stream engine, comprising:

[0036] A generation unit is configured to generate a logical flow graph of the received task to be processed, the logical flow graph including flow nodes representing each operator used to process the task to be processed.

[0037] The acquisition unit is used to acquire resource information of each flow node in the logical flow graph from the operator resource configuration;

[0038] The processing unit is used to merge flow nodes with the same set attributes in the logical flow graph into group nodes to obtain a logical group graph with resource information of each group node; wherein the resource information of the group node is obtained based on the resource information of the flow nodes in the group node; and submit the logical group graph to the resource scheduler; the resource scheduler is used to allocate resources for processing the task to be processed according to the logical group graph.

[0039] Optionally, the processing unit is further configured to:

[0040] Convert the aforementioned logic diagram into a task planning diagram;

[0041] Resource information of each group node is obtained from the logical group diagram, and resource information of each flow node is obtained from the logical flow diagram;

[0042] The resource information of each group of nodes and the resource information of each flow node are filled into the task plan graph, and the task plan graph is displayed.

[0043] Optionally, the processing unit is further configured to:

[0044] Receive user resource modification information from the task scheduler display interface;

[0045] Modify the corresponding operators in the operator resource configuration based on the resource modification information;

[0046] The logical group diagram is updated based on the modified operator resource configuration and sent to the resource scheduler.

[0047] Optionally, the generation unit is specifically used for:

[0048] The logical flow graph of the task to be processed is generated based on the FlinkJar or FlinkSql package.

[0049] Optionally, the processing unit is specifically used for:

[0050] The streaming engine client traverses each group of nodes in the task plan graph, maps each group of nodes in the task plan graph to each group of nodes in the logical group graph, and obtains the resource information of each group of nodes in the logical group graph.

[0051] The streaming engine client traverses each streaming node in the task schedule graph, maps each streaming node in the task schedule graph to a streaming node in the logical flow graph, and obtains the resource information of each streaming node in the logical flow graph. Optionally, the processing unit is specifically used for:

[0052] The streaming engine client obtains the resource modification information and the group node corresponding to the resource modification information, distributes the resource modification information equally according to the number of streaming nodes contained in the group node, and updates the resource modification information of the operator corresponding to the streaming node contained in the group node in the operator resource configuration.

[0053] Optionally, the processing unit is specifically used for:

[0054] The streaming engine client obtains the resource modification information and the streaming node corresponding to the resource modification information, and updates the resource modification information to the resource information of the operator corresponding to the streaming node in the operator resource configuration.

[0055] Thirdly, embodiments of the present invention also provide a computing device, comprising:

[0056] Memory, used to store computer programs;

[0057] The processor is configured to invoke a computer program stored in the memory and execute the resource setting method for the distributed streaming data stream engine as listed above, according to the obtained program.

[0058] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a computer-executable program, the computer-executable program being used to cause a computer to execute the resource setting method for a distributed streaming data stream engine listed in any of the above embodiments. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1A A schematic diagram of a system architecture for a Flink streaming engine provided in an embodiment of the present invention;

[0061] Figure 1B A schematic diagram of a system architecture for a resource scheduler provided in an embodiment of the present invention;

[0062] Figure 1C A schematic diagram of another system architecture for a resource scheduler provided in an embodiment of the present invention;

[0063] Figure 2 A flowchart illustrating a resource setting method applied to Flink, provided as an embodiment of the present invention;

[0064] Figure 3A A schematic diagram of a FlinkJar transformation logic flow diagram provided in an embodiment of the present invention;

[0065] Figure 3B A schematic diagram of a FlinkSql transformation logic flow graph provided in an embodiment of the present invention;

[0066] Figure 4A A schematic diagram of a logic flow graph generated using the FlinkJar programming language, provided as an embodiment of the present invention;

[0067] Figure 4B A schematic diagram of a logic group diagram provided for an embodiment of the present invention;

[0068] Figure 5 This is a schematic diagram illustrating a process for modifying operator resource configuration according to an embodiment of the present invention;

[0069] Figure 6A A schematic diagram of a task planning diagram provided for an embodiment of the present invention;

[0070] Figure 6B A schematic diagram of a task planning diagram provided for an embodiment of the present invention;

[0071] Figure 7 A schematic flowchart illustrating a method for setting resources in a distributed streaming data stream engine, provided by an embodiment of the present invention;

[0072] Figure 8 A schematic diagram of a resource setting device applied to a distributed streaming data stream engine provided in an embodiment of the present invention;

[0073] Figure 9 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0074] To make the objectives, implementation methods and advantages of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the described exemplary embodiments are only some embodiments of this application, and not all embodiments.

[0075] Based on the exemplary embodiments described in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the appended claims. Furthermore, although the disclosures in this application are presented by way of one or more exemplary examples, it should be understood that each aspect of these disclosures can also constitute a complete implementation on its own.

[0076] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0077] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities and do not necessarily imply a specific order or sequence, unless otherwise indicated. It should be understood that such terms can be used interchangeably where appropriate, for example, to implement the application in a sequence other than those given in the embodiments illustrated or described herein.

[0078] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclusively include, for example, a product or device that includes a series of components is not necessarily limited to those that are explicitly listed, but may include other components that are not explicitly listed or that are inherent to such product or device.

[0079] Figure 1A This paper exemplifies a system architecture for a Flink streaming engine applicable to an embodiment of the present invention, including a streaming engine client 100 and a resource scheduler 200. This is merely an illustrative description of the technical solution of this embodiment; in practice, a Flink streaming engine may contain many components.

[0080] The streaming engine client 100 receives tasks to be processed, divides them into multiple operators, and sends a series of configuration information, such as the parallelism of each operator, the total memory of each streaming engine timer (TM), and the number of slots on each TM, to the resource scheduler 200. The resource scheduler 200 allocates resources according to the configuration information.

[0081] Figure 1B An exemplary system architecture of a resource scheduler 200 provided in an embodiment of the present invention is illustrated. It includes a JM (JobManager) and multiple TMs (such as...). Figure 1B The TM1, TM2, and TM3 are used to execute specific tasks. Each TM contains multiple slots.

[0082] The following example illustrates the working process of the resource scheduler.

[0083] The streaming engine client divides the task to be processed into three operators: Operator 1 - Data Acquisition, Operator 2 - Maximum Value Determination, and Operator 3 - Output. Operator 1 has a parallelism of 2, Operator 2 has a parallelism of 2, and Operator 3 has a parallelism of 3. The Resource Scheduler (JM) receives the task to be processed and its configuration information, and then sends the three operators to the Transaction Manager (TM) for processing. For example, if Operator 1 is sent to TM1, and TM1 has 100GB of memory and contains three slots, then each slot will have approximately 33.3GB of resources allocated. Since Operator 1 has a parallelism of 2, TM1 will allocate two slots to execute Operator 1. Similarly, if Operator 2 is sent to TM2, and TM2 has 100GB of memory and contains three slots, then each slot will have approximately 33.3GB of resources allocated. Since Operator 2 has a parallelism of 2, TM2 will allocate two slots to execute Operator 2. Operator 3 is sent to TM3, which has 100GB of memory and contains 3 slots. Each slot receives approximately 33.3GB of resources. Operator 3 has a parallelism of 3, so TM3 allocates 3 slots to execute it. This is just an example; in actual implementation, different operators may be sent to the same TM for processing.

[0084] Figure 1C An exemplary illustration shows the workflow of another resource scheduler 200 provided in an embodiment of the present invention.

[0085] Using the example above, since operator 1 and operator 2 are adjacent and have the same degree of parallelism, they can be grouped into the same group and sent to the same processing unit (TM) for processing. For example, if sent to TM1, TM1 will allocate two slots to execute operator 1 first, and then execute operator 2 after operator 1 has finished. In this way, the serialization / deserialization / transmission overhead required for the execution result of operator 1 to flow between different TMs can be avoided.

[0086] It can be observed that in the two workflows of the resource scheduler described above, since the resources required for each operator are not determined, the resource scheduler can only distribute the resources of each TM equally, allocating the number of slots according to the parallelism of each operator. This may lead to the following situation: Since the computational logic of operators is different, and different logics consume different resources, it is obviously unreasonable to allocate the same resources to operators with different logics. If too many resources are allocated, there will be a waste of resources and the operation resources of other tasks will be squeezed out; if too few resources are allocated, the execution speed of tasks will slow down, leading to system crashes.

[0087] Therefore, if the resources for task operators can be configured in a targeted, fine-grained manner, the resource scheduler can allocate resources of the appropriate size, thereby improving resource utilization. The streaming engine client receives tasks to be processed, divides them into different operators, and sets the required resources for each operator, for example: Operator 1 - data acquisition - parallelism 2 - 5GB memory. Then, after the resource scheduler receives the tasks with the configured resources, it can allocate slots of the appropriate size for computation based on the resource requests. For example, the resource scheduler sends Operator 1 to TM1 for computation, and TM1 allocates two slots totaling 5GB to process Operator 1.

[0088] However, existing Flink streaming engine clients typically use the Flink JAR or Flink SQL programming languages. Flink SQL does not support fine-grained configuration of operator resources; it only allows configuration based on native Flink settings, such as total memory for the Streaming Manager (TM), parallelism, and the number of slots per TM. It cannot perform fine-grained resource configurations, such as the number of CPUs or memory required for each operator. While Flink JAR supports operator resource settings, these settings are hardcoded into the code and cannot be flexibly modified. If there is a sudden surge in workload, the previously configured operator resources may be insufficient to support such a large workload, leading to system crashes. Modifying operator resource configurations requires rewriting the program and repackaging, a very cumbersome process. Therefore, current Flink applications implemented based on these two APIs do not flexibly support fine-grained resource settings and adjustments.

[0089] This invention provides a resource configuration method for Flink, such as... Figure 2 As shown, it includes:

[0090] Step 201: The stream engine client generates a logical flow graph of the task to be processed based on the received task to be processed. The logical flow graph includes each flow node representing each operator used to process the task to be processed.

[0091] Step 202: The stream engine client obtains the resource information of each stream node in the logical flow graph from the operator resource configuration;

[0092] Step 203: The flow engine client merges flow nodes with the same set attributes in the logical flow graph into group nodes to obtain a logical group graph with resource information of each group node; wherein, the resource information of the group node is obtained based on the resource information of the flow nodes in the group node;

[0093] Step 204: The streaming engine client submits the logical graph to the resource scheduler; the resource scheduler allocates resources for processing the pending task according to the logical graph.

[0094] In step 201, the streaming engine client generates a logical flow graph (StreamGraph) for the received tasks. The logical flow graph is an initial graph generated from code written by the user via the Stream API, used to represent the program's topology. Different programming languages ​​use different methods to generate logical flow graphs.

[0095] If FlinkJar's programming language is used, FlinkJar will transform the logic flow diagram as follows: Figure 3A As shown: A PackagedProgram object is constructed based on the Flink API using the basic properties of FlinkJar, such as the Jar package, main class, and configuration information. Then, the Pipeline object is obtained by calling the static method PackagedProgramUtils.getPipelineFromProgram. We only focus on streaming tasks, and Pipeline is equivalent to StreamGraph.

[0096] If FlinkSql is used as the programming language, the FlinkSql transformation logic flow diagram is as follows: Figure 3B As shown: Parse the SQL text according to the semicolon ";" to obtain a set of SQL statements. Construct a context environment StreamExecutionEnvironment, process each SQL statement in the context environment to apply the SQL logic to the context, and obtain the StreamGraph through StreamExecutionEnvironment.getStreamGraph.

[0097] The above are merely examples. The embodiments of the present invention do not limit the programming language and methods used to generate the logic flow diagram of the task to be processed.

[0098] The logic flow diagram illustrates the settings of the streaming engine client for the tasks to be processed. Figure 4A This illustrates a possible structure for generating logic flow graphs using the FlinkJar programming language, such as... Figure 4A As shown, the task to be processed is divided into 3 operators, i.e., 3 stream nodes, and an ID is generated for each stream node. Parallelism is set for each stream node, along with some fine-grained settings, such as the number of CPUs required for each stream node, on-heap memory, and off-heap memory. These settings are hardcoded into the FlinkJar program and cannot be flexibly changed. If the logic flow graph is generated using the FlinkSql programming language, these fine-grained settings will not be present; only the ID, name, and parallelism will be included.

[0099] In step 202, the flow engine client obtains the resource information of each flow node in the logical flow graph from the operator resource configuration.

[0100] Operator resource configurations are stored in a database or other storage media, and users can flexibly modify them. The specific modification methods are detailed below; the process of generating operator resource configurations is introduced first.

[0101] The logical flow graph generated in step 201 contains the name and ID of each flow node. Then, the flow engine client saves the mapping relationship between the name and ID of each flow node in the database, thus generating the operator resource configuration. Table 1 shows a possible structure of the operator resource configuration. As shown in Table 1, the correspondence between operators, IDs, and various resource information can be determined.

[0102] Table 1

[0103] ID Parallelism Number of CPUs Heap memory (G) Off-heap memory (G) Operator 1 name A 2 1 1 1 Operator 2 name B 2 1 1 1 Operator 3 Name C 3 1 1 1

[0104] Since developers can flexibly modify operator resource configurations, and each modification updates the operator resource configuration, in step 202, after each developer modifies the operator resource configuration, the stream engine client can obtain the developer's resource modification information from the operator resource configuration and then set the resource modification information in the resource information of the stream node with the same ID in the logical flow graph. For example, if a developer changes the CPU count of the operator with ID A in the operator resource configuration to 2, then the CPU count of the stream node corresponding to the operator with ID A in the logical flow graph will also be updated to 2. If the developer does not modify the operator resource configuration, the logical flow graph will not be updated.

[0105] In step 203, the flow engine client merges flow nodes with the same set attributes in the logical flow graph into group nodes to obtain a logical group graph with resource information of each group node; wherein, the resource information of the group node is obtained based on the resource information of the flow nodes in the group node.

[0106] The same attribute settings can be used for adjacent stream nodes with the same degree of parallelism, for example, in Figure 4A In the structure of the shown logical flow graph, flow node 1 and flow node 2 are adjacent and have the same degree of parallelism, so they are merged into a group node, thus obtaining a logical group graph JobGraph containing resource information of each group node. For example... Figure 4B As shown. In reality, the logic diagram exists in code form, but for ease of understanding, it is presented as a diagram here. The logic diagram contains a great deal of information, not just... Figure 4BThe information listed is for illustrative purposes only. Flow node 1 and flow node 2 are merged into group node 1. The resource information of the group node is obtained based on the resource information of each of the included flow nodes. The method for generating the logical group graph is not limited. If using the programming languages ​​FlinkSql or FlinkJar, the logical flow graph is converted into a logical group graph using getJobGraph.

[0107] In step 204, the logical group diagram is submitted to the resource scheduler. Since the logical group diagram contains fine-grained resource configuration information for each group node, the resource scheduler will allocate resources to the tasks to be processed based on the resource information in the logical group diagram.

[0108] For example, when developers will Figure 4B Once the logical group diagram is submitted to the resource scheduler, the resource scheduler allocates two slots in TM1 to group node 1 for computation. The total number of CPUs in the two slots is 2, the total on-heap memory is 2, and the total off-heap memory is 2.

[0109] It can be observed that by using the resource setting method provided in this embodiment of the invention, fine-grained resource settings for the resources required by the operators of a task can be easily and quickly achieved through operator resource configuration. Furthermore, the resource information of each flow node in the logical flow graph is updated accordingly after each operator resource configuration update. This allows for convenient setting of the resource information of each flow node in the logical flow graph as needed. Further, based on the updated logical flow graph, the resource information of each group node in the generated logical group graph is also updated accordingly. When the logical group graph configured according to the developer's needs is submitted to the resource scheduler, the resource scheduler will allocate corresponding resources based on the resource settings of each flow node in the logical group graph. Thus, when data traffic or system network speed changes, flexible and fine-grained control of the resource information of each flow node and group node can be achieved, improving the efficiency of resource setting. Subsequently, the Yarn cluster can allocate resources of appropriate size according to the resources configured by the developer on the client, thereby improving resource utilization.

[0110] The following details the process by which developers modify operator resource configurations. For example... Figure 5 As shown, it includes:

[0111] Step 501: The stream engine client converts the logical group diagram into a task plan diagram.

[0112] The logic diagram is actually presented in code form, and it contains far more information than that. Figure 4BThe information shown extracts key information from the logic group diagram to generate a task plan diagram for developers. The task plan diagram also displays runtime information, allowing developers to assess the task's operational logic. It's worth noting that in existing technologies, regardless of whether FlinkSQL or FlinkJar is used, some fine-grained resource information is removed during the conversion of the logic group diagram into a task plan diagram. Therefore, the resulting task plan diagram may appear... Figure 6A As shown. To enable users to view and modify fine-grained resource information in the task plan diagram, the following steps are performed.

[0113] Step 502: The stream engine client obtains the resource information of each group node from the logical group graph and the resource information of each stream node from the logical flow graph.

[0114] The logical group diagram contains resource information for each group of nodes, and the logical flow diagram contains resource information for each flow node. Therefore, the flow engine client can obtain resource information for each group of nodes from the logical group diagram and resource information for each flow node from the logical flow diagram. Specific acquisition methods may include the following:

[0115] Method 1: Traverse each group of nodes in the task plan graph and obtain the resource information of the group nodes with the same ID in the logical group graph; traverse each flow node in the task plan graph and obtain the resource information of the flow nodes with the same ID in the logical flow graph.

[0116] Method 2: Since the operator resource configuration contains the resource information of each operator, i.e. each flow node, we can traverse each flow node in the task plan graph and obtain the resource information of flow nodes with the same ID in the operator resource configuration; as for the resource information of group nodes, it can still be obtained from the logical group graph.

[0117] Step 503: The streaming engine client fills the task schedule graph with the resource information of each group of nodes and the resource information of each streaming node, and then displays the task schedule graph. For example... Figure 6B As shown.

[0118] The resulting task plan graph displays resource information for each flow node and each group of nodes, visually presented to developers. Developers can modify the resource information based on task performance. For example, if a developer notices a significant decrease in the running speed of a task, they can modify the resource information, such as increasing the number of CPUs, increasing on-heap memory, or increasing off-heap memory.

[0119] Step 504: The streaming engine client receives the user's resource modification information from the task scheduler display interface.

[0120] For example, when a developer clicks on flow node 1 in the task schedule graph, they can set the resource information of flow node 1, such as setting the heap memory to 3G. Then, the flow engine client receives the resource modification information from the developer's interface in the task schedule graph. The resource modification information includes setting the heap memory to 3G and the ID of the flow node corresponding to the resource modification information.

[0121] Step 505: The stream engine client modifies the corresponding operator in the operator resource configuration based on the resource modification information.

[0122] For example, the streaming engine client sets the heap memory of streaming node 1 in the operator resource configuration to 3G.

[0123] Step 506: The stream engine client updates the logical group diagram based on the modified operator resource configuration and sends it to the resource scheduler.

[0124] After the operator resource configuration is updated, the logical flow graph will obtain the updated information and generate a logical group graph based on the updated resource information. The updated logical group graph is then submitted to the resource scheduler, which will allocate resources according to the latest submitted resource information of each group of nodes.

[0125] Developers can make various modifications to the resource information of flow nodes and group nodes as needed. Two possible implementations are listed below.

[0126] Example 1

[0127] Will as Figure 6B The task plan diagram is shown. The developer clicks on group node 1, modifies the heap memory of the group node to 5G, and clicks OK. Since group node 1 contains two stream nodes, the two stream nodes will evenly divide the heap memory of the group node, so the heap memory of each stream node is 2.5G. In the operator resource configuration, the heap memory of the operators corresponding to the two stream nodes in group node 1 is updated to 2.5G.

[0128] For example, if the structure of the operator resources before configuration is as shown in Table 1, then the updated operator resources are as shown in Table 2.

[0129] Table 2

[0130] ID Parallelism Number of CPUs Heap memory (G) Off-heap memory (G) Operator 1 name A 2 1 2.5 1 Operator 2 name B 2 1 2.5 1 Operator 3 Name C 3 1 1 1

[0131] Example 2

[0132] You can also directly modify the resource information of the stream node.

[0133] Will as Figure 6BThe task schedule diagram is shown. The developer clicks on stream node 1, modifies the stream node's heap memory to 3G, and clicks OK. The stream engine client then obtains the resource modification information and determines the ID of stream node 1 corresponding to the resource modification information. In the operator resource configuration, the resource information of the operator with the same ID as this stream node is changed to 3G.

[0134] For example, if the structure of the operator resources before configuration is as shown in Table 2, then the updated operator resources are as shown in Table 3.

[0135] Table 3

[0136] ID Parallelism Number of CPUs Heap memory (G) Off-heap memory (G) Operator 1 name A 2 1 3 1 Operator 2 name B 2 1 2.5 1 Operator 3 Name C 3 1 1 1

[0137] Developers can make various changes to the resource information of each flow node and each group node as needed, and each change to the resource information will overwrite the previous change.

[0138] To better explain the embodiments of the present invention, the resource setting process applied to the distributed streaming data stream engine will be described below in a specific implementation scenario. For example... Figure 7 As shown. The order of the steps is only for illustrative purposes; the order can be adjusted arbitrarily in actual implementation.

[0139] Step 701: The stream engine client generates a logical flow graph of the tasks to be processed based on the received tasks.

[0140] Step 702: Developers set the resource information of flow nodes or group nodes based on the task plan graph display interface and generate resource modification information.

[0141] Step 703: The stream engine client modifies the corresponding operator in the operator resource configuration based on the resource modification information.

[0142] Step 704: The stream engine client obtains the resource information of each stream node in the logical flow graph from the operator resource configuration.

[0143] Step 705: The flow engine client merges flow nodes with the same set attributes in the logical flow graph into group nodes to obtain a logical group graph with resource information of each group node.

[0144] Step 706: The stream engine client submits the logical group diagram to the resource scheduler.

[0145] Step 707: The stream engine client converts the logical group diagram into a task plan diagram.

[0146] Step 708: The streaming engine client obtains the resource information of each group node from the logical group graph and the resource information of each streaming node from the logical flow graph; the streaming engine client fills the resource information of each group node and the resource information of each streaming node into the task plan graph and displays the task plan graph.

[0147] Based on the same technological concept Figure 8 An exemplary embodiment of the present invention illustrates the structure of a resource setting device for a distributed streaming data stream engine, which can execute a resource setting process for the distributed streaming data stream engine.

[0148] like Figure 8 As shown, the device specifically includes:

[0149] The generation unit 801 is used to generate a logical flow graph of the received task to be processed, the logical flow graph including flow nodes representing each operator used to process the task to be processed.

[0150] The acquisition unit 802 is used to acquire the resource information of each flow node in the logical flow graph from the operator resource configuration;

[0151] Processing unit 803 is used to merge flow nodes with the same set attributes in the logical flow graph into group nodes to obtain a logical group graph with resource information of each group node; wherein, the resource information of the group node is obtained based on the resource information of the flow nodes in the group node; and submit the logical group graph to the resource scheduler; the resource scheduler is used to allocate resources for processing the task to be processed according to the logical group graph.

[0152] Optionally, the processing unit 803 is further configured to:

[0153] Convert the aforementioned logic diagram into a task planning diagram;

[0154] Resource information of each group node is obtained from the logical group diagram, and resource information of each flow node is obtained from the logical flow diagram;

[0155] The resource information of each group of nodes and the resource information of each flow node are filled into the task plan graph, and the task plan graph is displayed.

[0156] Optionally, the processing unit 803 is further configured to:

[0157] Receive user resource modification information from the task scheduler display interface;

[0158] Modify the corresponding operators in the operator resource configuration based on the resource modification information;

[0159] The logical group diagram is updated based on the modified operator resource configuration and sent to the resource scheduler.

[0160] Optionally, the generation unit 801 is specifically used for:

[0161] The logical flow graph of the task to be processed is generated based on the FlinkJar or FlinkSql package.

[0162] Optionally, the processing unit 803 is specifically used for:

[0163] The streaming engine client traverses each group of nodes in the task plan graph, maps each group of nodes in the task plan graph to each group of nodes in the logical group graph, and obtains the resource information of each group of nodes in the logical group graph.

[0164] The streaming engine client traverses each streaming node in the task plan graph, maps each streaming node in the task plan graph to each streaming node in the logical flow graph, and obtains the resource information of each streaming node in the logical flow graph.

[0165] Optionally, the processing unit 803 is specifically used for:

[0166] The streaming engine client obtains the resource modification information and the group node corresponding to the resource modification information, distributes the resource modification information equally according to the number of streaming nodes contained in the group node, and updates the resource modification information of the operator corresponding to the streaming node contained in the group node in the operator resource configuration.

[0167] Optionally, the processing unit 803 is specifically used for:

[0168] The streaming engine client obtains the resource modification information and the streaming node corresponding to the resource modification information, and updates the resource modification information to the resource information of the operator corresponding to the streaming node in the operator resource configuration.

[0169] Based on the same technical concept, embodiments of this application provide a computer device, such as... Figure 9 As shown, it includes at least one processor 901 and a memory 902 connected to at least one processor. In this embodiment, the specific connection medium between the processor 901 and the memory 902 is not limited. Figure 9 Taking the connection between processor 901 and memory 902 via a bus as an example, the bus can be divided into address bus, data bus, control bus, etc.

[0170] In this embodiment of the application, the memory 902 stores instructions that can be executed by at least one processor 901. By executing the instructions stored in the memory 902, at least one processor 901 can perform the steps of the resource setting method applied to the distributed streaming data stream engine described above.

[0171] The processor 901 is the control center of the computer device, capable of connecting various parts of the device via various interfaces and lines. It performs resource configuration by running or executing instructions stored in the memory 902 and accessing data stored in the memory 902. Optionally, the processor 901 may include one or more processing units. The processor 901 may integrate an application processor and a modem processor. The application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may not be integrated into the processor 901. In some embodiments, the processor 901 and the memory 902 may be implemented on the same chip; in other embodiments, they may be implemented on separate chips.

[0172] The processor 901 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0173] Memory 902, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 902 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 902 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. In the embodiments of this application, memory 902 can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0174] Based on the same technical concept, embodiments of the present invention also provide a computer-readable storage medium storing a computer-executable program, the computer-executable program being used to cause a computer to perform the resource setting method for a distributed streaming data stream engine listed in any of the above methods.

[0175] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0176] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0177] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0178] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0179] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A resource setting method applied to a distributed streaming data stream engine, characterized in that, include: The stream engine client generates a logical flow graph of the received task to be processed, which includes flow nodes representing operators used to process the task to be processed. The stream engine client obtains the resource information of each stream node in the logical flow graph from the operator resource configuration; The resource information includes: parallelism, number of CPUs, on-heap memory, and off-heap memory; The streaming engine client merges streaming nodes with the same set attributes in the logical streaming graph into group nodes, resulting in a logical group graph with resource information for each group node; wherein, the resource information of the group node is obtained based on the resource information of the streaming nodes in the group node; The streaming engine client converts the logical group diagram into a task plan diagram, which is used to visualize and display the fine-grained resource information for users and / or developers to view, and to make various changes to the resource information of each streaming node and each group node as needed; The streaming engine client submits the logical graph to the resource scheduler; the resource scheduler allocates resources for processing the pending tasks according to the logical graph.

2. The method as described in claim 1, characterized in that, The method further includes: The streaming engine client obtains resource information of each group node from the logical group graph and resource information of each streaming node from the logical flow graph; The streaming engine client fills the task schedule graph with the resource information of each group of nodes and the resource information of each streaming node, and then displays the task schedule graph.

3. The method as described in claim 2, characterized in that, The method further includes: The streaming engine client receives resource modification information from the user's resource plan display interface; The streaming engine client modifies the corresponding operator in the operator resource configuration based on the resource modification information; The stream engine client updates the logical graph based on the modified operator resource configuration and sends it to the resource scheduler.

4. The method as described in claim 1, characterized in that, The streaming engine client generates a logical flow graph of the received tasks based on the tasks to be processed, including: The streaming engine client generates the logical flow graph of the task to be processed based on the FlinkJar or FlinkSql package.

5. The method as described in claim 2, characterized in that, The streaming engine client obtains resource information of each group node from the logical group graph and resource information of each streaming node from the logical flow graph, including: The streaming engine client traverses each group of nodes in the task plan graph, maps each group of nodes in the task plan graph to each group of nodes in the logical group graph, and obtains the resource information of each group of nodes in the logical group graph. The streaming engine client traverses each streaming node in the task plan graph, maps each streaming node in the task plan graph to each streaming node in the logical flow graph, and obtains the resource information of each streaming node in the logical flow graph.

6. The method as described in claim 3, characterized in that, The streaming engine client modifies the corresponding operator in the operator resource configuration based on the resource modification information, including: The streaming engine client obtains the resource modification information and the group node corresponding to the resource modification information, distributes the resource modification information equally according to the number of streaming nodes contained in the group node, and updates the resource modification information of the operator corresponding to the streaming node contained in the group node in the operator resource configuration.

7. The method as described in claim 3, characterized in that, The streaming engine client modifies the corresponding operator in the operator resource configuration based on the resource modification information, including: The streaming engine client obtains the resource modification information and the streaming node corresponding to the resource modification information, and updates the resource modification information to the resource information of the operator corresponding to the streaming node in the operator resource configuration.

8. A resource setting device for a distributed streaming data stream engine, characterized in that, include: A generation unit is configured to generate a logical flow graph of the received task to be processed, the logical flow graph including flow nodes representing each operator used to process the task to be processed. The acquisition unit is used to acquire resource information of each flow node in the logical flow graph from the operator resource configuration; The resource information includes: parallelism, number of CPUs, on-heap memory, and off-heap memory; The processing unit is used to merge flow nodes with the same set attributes in the logical flow graph into group nodes to obtain a logical group graph with resource information of each group node; wherein, the resource information of the group node is obtained based on the resource information of the flow nodes in the group node; The streaming engine client converts the logical group diagram into a task plan diagram, which is used to visualize and display the fine-grained resource information for users and / or developers to view, and to make various changes to the resource information of each streaming node and each group node as needed; The logical graph is submitted to the resource scheduler; the resource scheduler is used to allocate resources for processing the task to be processed according to the logical graph.

9. A computing device, characterized in that, include: Memory, used to store computer programs; A processor is configured to invoke a computer program stored in the memory and execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer-executable program for causing a computer to perform the method according to any one of claims 1 to 7.

Images