Task evaluation method and device based on dynamic expansion of Flink engine computing nodes

Abstract

The present application proposes a task evaluation method based on the dynamic expansion of computing nodes in the Flink engine, including: receiving tasks sent by the client; receiving node expansion instructions sent by the client when data hotspots appear, modifying the topology of the execution graph of the task according to the node expansion instructions, and notifying the upstream operator task of the target operator task to add result partitions and change virtual hash mapping tables and virtual routing tables, wherein the target operator task is the operator task where data hotspots appear; starting the newly added computing node through the task manager, initializing the input gateway and result partition of the newly added computing node; notifying the downstream operator task of the target operator task to add input channels; processing user data according to the topology of the modified execution graph of the task through the task manager to obtain the task evaluation result. The present application solves the problem of data skew when the Flink engine processes data, improves the data calculation speed of the system, and meets the real-time scenario requirements of tasks.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a task evaluation method and device based on the dynamic expansion of Flink engine computing nodes. Background Art

[0002] With the rapid development of big data technology, enterprises in the field of data processing have higher requirements for the speed and stability of computing engines in processing data. As a new generation of stream processing engine, Flink has become the mainstream in the field of real-time computing. At the same time, with the development of the Internet, credit data is growing exponentially. For a credit system with millions of users, PB-level real-time data needs to be processed every day. However, when using the Flink engine to process credit data, with the increase of system users and the increase of different types of data, when Flink uses the hash partitioning strategy to process credit data, sometimes a large amount of data is assigned to one node for processing, resulting in data hotspots on the node, which reduces the data calculation speed and throughput, and cannot meet the real-time scenario requirements of the task. Summary of the invention

[0003] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0004] To this end, the first purpose of this application is to propose a task evaluation method based on the dynamic expansion of Flink engine computing nodes. By dynamically expanding the computing nodes when data hotspots appear, the correct operation of the system is not affected. At the same time, the topological relationship between the original computing nodes is not affected, and the possibility of re-consuming data is avoided. It fundamentally solves the problem of data skew when the Flink engine processes data, improves the system's data calculation speed, and meets the real-time scenario requirements of the task.

[0005] The second objective of the present application is to propose a task evaluation device based on dynamic expansion of Flink engine computing nodes.

[0006] A third object of the present application is to provide a non-transitory computer-readable storage medium.

[0007] To achieve the above-mentioned purpose, the first embodiment of the present application proposes a task evaluation method based on dynamic expansion of Flink engine computing nodes, including: receiving a task sent by a client, wherein the task is generated by the client based on user data; receiving a node expansion instruction sent by the client when a data hotspot appears, modifying the topological structure of the execution graph of the task according to the node expansion instruction, and notifying the upstream operator task of the target operator task to add a result partition and change the virtual hash mapping table and the virtual routing table, wherein the target operator task is the operator task where the data hotspot appears; starting a new computing node through a task manager, initializing the input gateway and result partition of the new computing node, wherein the input gateway of the new computing node is connected to the result partition added by the upstream operator task; notifying the downstream operator task of the target operator task to add an input channel, wherein the input channel added by the downstream operator task is connected to the result partition of the new computing node; processing the user data through the task manager according to the topological structure of the modified execution graph of the task to obtain a task evaluation result.

[0008] Optionally, in one embodiment of the present application, the user data includes: one or more of the user's basic information, consumption data, loan data, income data, and social information.

[0009] Optionally, in one embodiment of the present application, the task manager processes the user data according to the topological structure of the modified execution graph of the task to obtain a task evaluation result, including:

[0010] The task manager transmits the user data processed by the upstream operator task to the input gateway of the target operator task through the result partition, wherein the result partition added by the upstream operator task is connected to the input gateway of the newly added computing node;

[0011] The user data received by the input gateway of the target operator task is divided into the N original computing nodes and the newly added computing nodes of the target operator task for processing, and the user data processed by the target operator task is transmitted to the input gateway of the downstream operator task through the result partition, wherein the result partition of the newly added computing node is connected to the input channel added by the downstream operator task;

[0012] The user data received by the input gateway of the downstream operator task is divided into M computing nodes of the downstream operator task for processing, and the user data processed by the downstream operator task is transmitted to the next operator task for further processing;

[0013] Wherein, N and M are positive integers.

[0014] Optionally, in one embodiment of the present application, processing the user data according to the topological structure of the modified execution graph of the task to obtain the task evaluation result further includes:

[0015] Pull user data through the initial operator task and transfer it to the next operator task for processing according to the topology of the modified task's execution graph;

[0016] The processed user data is output through the output operator task to obtain the task evaluation result.

[0017] Optionally, in one embodiment of the present application, changing the virtual hash mapping table and the virtual routing table includes:

[0018] The upstream operator task adds a mapping of the newly created result partition in the virtual hash mapping table, and divides the virtual hash value corresponding to the result partition of the upstream operator task into half to the newly created result partition;

[0019] According to the virtual hash mapping table, all virtual hash values ​​corresponding to the newly created result partition are mapped to the newly created result partition in the virtual routing table.

[0020] To achieve the above-mentioned purpose, the second embodiment of the present application proposes a task evaluation device based on dynamic expansion of Flink engine computing nodes, including: a job manager, which is used to receive tasks sent by a client, and receive node expansion instructions sent by the client when a data hotspot appears, modify the topological structure of the execution graph of the task according to the node expansion instruction, and notify the upstream operator task of the target operator task to add result partitions and change virtual hash mapping tables and virtual routing tables, wherein the target operator task is the operator task where the data hotspot appears; notify the task manager to start a new computing node; notify the downstream operator task of the target operator task to add an input channel, wherein the input channel object added by the downstream operator task is connected to the result partition of the new computing node; process the user data according to the topological structure of the modified execution graph of the task through the task manager to obtain a task evaluation result;

[0021] The task manager is used to receive the instruction sent by the job manager, start the newly added computing node, initialize the input gateway and result partition of the newly added computing node, wherein the input gateway of the newly added computing node is connected to the result partition added by the upstream operator task; and is also used to process the user data according to the topological structure of the execution graph of the modified task to obtain the task evaluation result;

[0022] The client is used to build tasks, send tasks to the job manager, and send node expansion instructions to the job manager when data hotspots appear.

[0023] In order to achieve the above objectives, the third aspect embodiment of the present application proposes a non-temporary computer-readable storage medium. When the instructions in the storage medium are executed by a processor, a task evaluation method based on dynamic expansion of Flink engine computing nodes can be executed.

[0024] The task evaluation method based on dynamic expansion of Flink engine computing nodes, the task evaluation device based on dynamic expansion of Flink engine computing nodes, and the non-temporary computer-readable storage medium of the embodiments of the present application dynamically expand computing nodes when data hotspots appear, thereby not affecting the correct operation of the system, and at the same time not affecting the topological relationship between the original computing nodes, avoiding the possibility of re-consuming data, fundamentally solving the problem of data skew when the Flink engine processes data, improving the data calculation speed of the system, and meeting the real-time scenario requirements of tasks.

[0025] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0027] Figure 1 A flowchart of a task evaluation method based on dynamic expansion of Flink engine computing nodes provided in Example 1 of the present application;

[0028] Figure 2 A schematic diagram of node dynamic expansion of a task evaluation method based on dynamic expansion of Flink engine computing nodes according to an embodiment of the present application;

[0029] Figure 3 It is a task physical execution diagram of an embodiment of the present application;

[0030] Figure 4 It is a physical execution graph of tasks after node expansion of a task evaluation method based on dynamic expansion of Flink engine computing nodes in an embodiment of the present application;

[0031] Figure 5 A schematic diagram of a consistent hashing algorithm ring of a task evaluation method based on dynamic expansion of Flink engine computing nodes according to an embodiment of the present application;

[0032] Figure 6 A schematic diagram of a virtual hash mapping table change process of a task evaluation method based on dynamic expansion of Flink engine computing nodes according to an embodiment of the present application;

[0033] Figure 7 A schematic diagram of a virtual routing table change process of a task evaluation method based on dynamic expansion of Flink engine computing nodes according to an embodiment of the present application;

[0034] Figure 8A structural diagram of a task evaluation device based on dynamic expansion of Flink engine computing nodes provided in Example 2 of the present application. DETAILED DESCRIPTION

[0035] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0036] When using Flink to process data, if some nodes use a hashing strategy based on user IDs for data partitioning, there is a probability that a large amount of data will be sent to a subtask node for calculation. At this time, data hotspots will appear in the subtask nodes of some nodes. There are generally two solutions to solve data hotspots:

[0037] (1) Manually reset the node parallelism. This method directly stops the program and resets the task parallelism to a reasonable level based on the data hash characteristics to avoid data hot spots. It has good stability effects in scenarios such as processing bounded data and clarifying data hash characteristics.

[0038] (2) Random prefix method. This method first adds different random numbers to the same hash keyword prefix to reduce the capacity of a single computing node, then deletes the prefix of the key and obtains the final result through global aggregation. This method is flexible and achieves good performance in many scenarios.

[0039] Disadvantages of existing solutions: For solution (1), this solution cannot be applied to the field of stream computing. First, since stream computing applications are long-term applications, if the program is stopped directly, it will cause temporary unavailability of the program, which is very fatal for businesses with high availability requirements; second, after resetting a reasonable degree of parallelism, the program only temporarily solves the data hotspot problem. As the number of users continues to grow, the data hotspot problem may reappear. At this time, it may be necessary to stop the program and set a reasonable degree of parallelism again. It can be seen that this manually set degree of parallelism does not fundamentally solve the scalability problem of data hotspot nodes. For solution (2), first, the addition of random numbers causes the subsequent global aggregation to require additional read costs (the relationship between random numbers and keywords needs to be maintained). Second, the state of stateful computing nodes cannot be maintained.

[0040] This application proposes a method for dynamically expanding computing nodes in the Flink engine. It adopts the idea of ​​consistency and dynamic, establishes virtual nodes in advance (virtual nodes do not exist, they are just a mark), and then maintains the mapping relationship between virtual nodes and actual physical nodes (here refers to the slot logical resources in Flink, not physical machines) in the nodes. The mapping algorithm uses an equal division algorithm. For nodes with data skew, half of the virtual nodes are directly divided to the newly added nodes. Since the number of virtual nodes is 2^32, in theory, the nodes can be adaptively and infinitely expanded, fundamentally solving the problem of insufficient computing node capacity in the stream computing system.

[0041] The present application proposes a method for dynamically expanding computing nodes of a Flink engine. On the one hand, by taking the advantages of the dynamic expansion computing node mode, the stability of the program is ensured, while avoiding the problem of inaccurate parallelism setting caused by manual intervention. On the other hand, the consistent hash partitioning strategy greatly reduces the dispersion of data with the same key value after partitioning, which not only quickly avoids the problem of data skew in the program, but also does not affect the correctness of the program due to the change in the topology structure.

[0042] The following describes a task evaluation method and device based on dynamic expansion of Flink engine computing nodes in an embodiment of the present application with reference to the accompanying drawings.

[0043] Figure 1 This is a flowchart of a task evaluation method based on dynamic expansion of Flink engine computing nodes provided in Example 1 of the present application.

[0044] like Figure 1 As shown, the task evaluation method based on the dynamic expansion of Flink engine computing nodes includes the following steps:

[0045] Step 101, receiving a task sent by a client, wherein the task is generated by the client according to user data;

[0046] Step 102, receiving a node expansion instruction sent by the client when a data hotspot appears, modifying the topological structure of the execution graph of the task according to the node expansion instruction, and notifying the upstream operator task of the target operator task to add a result partition and change the virtual hash mapping table and the virtual routing table, wherein the target operator task is the operator task where the data hotspot appears;

[0047] Step 103, starting the newly added computing node through the task manager, initializing the input gateway and result partition of the newly added computing node, wherein the input gateway of the newly added computing node is connected to the result partition added by the upstream operator task;

[0048] Step 104, notifying the downstream operator task of the target operator task to add an input channel, wherein the input channel added by the downstream operator task is connected to the result partition of the newly added computing node;

[0049] Step 105 , the task manager processes the user data according to the modified topological structure of the execution graph of the task to obtain a task evaluation result.

[0050] The task evaluation method based on the dynamic expansion of the computing node of the Flink engine of the embodiment of the present application receives the task sent by the client, wherein the task is generated by the client according to the user data; receives the node expansion instruction sent by the client when the data hotspot appears, modifies the topological structure of the execution graph of the task according to the node expansion instruction, and notifies the upstream operator task of the target operator task to add the result partition and change the virtual hash mapping table and the virtual routing table, wherein the target operator task is the operator task where the data hotspot appears; starts the newly added computing node through the task manager, initializes the input gateway and result partition of the newly added computing node, wherein the input gateway of the newly added computing node is connected to the result partition added by the upstream operator task; notifies the downstream operator task of the target operator task to add the input channel, wherein the input channel added by the downstream operator task is connected to the result partition of the newly added computing node; processes the user data according to the topological structure of the execution graph of the modified task through the task manager to obtain the task evaluation result. As a result, the computing nodes can be dynamically expanded when data hotspots appear without affecting the correct operation of the system. At the same time, the topological relationship between the original computing nodes is not affected, avoiding the possibility of re-consuming data. This fundamentally solves the problem of data skew when the Flink engine processes data, improves the system's data calculation speed, and meets the real-time scenario requirements of the task.

[0051] Furthermore, in an embodiment of the present application, user data includes: one or more of the user's basic information, consumption data, loan data, income data, and social information.

[0052] The user data may be credit data in a credit assessment scenario in the credit investigation field. Before performing a credit task assessment, the user data is first obtained, and then the data may be pre-processed using a big data processing framework (processing tools such as Flink and Spark).

[0053] Furthermore, in the embodiment of the present application, the task manager processes the user data according to the topological structure of the modified execution graph of the task to obtain the task evaluation result, including:

[0054] The task manager transmits the user data processed by the upstream operator task to the input gateway of the target operator task through the result partition, wherein the result partition added by the upstream operator task is connected to the input gateway of the newly added computing node;

[0055] The user data received by the input gateway of the target operator task is divided into the N original computing nodes and the newly added computing nodes of the target operator task for processing, and the user data processed by the target operator task is transmitted to the input gateway of the downstream operator task through the result partition, wherein the result partition of the newly added computing node is connected to the input channel added by the downstream operator task;

[0056] The user data received by the input gateway of the downstream operator task is divided into M computing nodes of the downstream operator task for processing, and the user data processed by the downstream operator task is transmitted to the next operator task for further processing;

[0057] Wherein, N and M are positive integers.

[0058] Furthermore, in the embodiment of the present application, the user data is processed according to the topological structure of the modified execution graph of the task to obtain the task evaluation result, which also includes:

[0059] Pull user data through the initial operator task and transfer it to the next operator task for processing according to the topology of the modified task's execution graph;

[0060] The processed user data is output through the output operator task to obtain the task evaluation result.

[0061] Furthermore, in an embodiment of the present application, changing the virtual hash mapping table and the virtual routing table includes:

[0062] The upstream operator task adds a mapping of the newly created result partition in the virtual hash mapping table, and divides the virtual hash value corresponding to the result partition of the upstream operator task into half to the newly created result partition;

[0063] According to the virtual hash mapping table, all virtual hash values ​​corresponding to the newly created result partition are mapped to the newly created result partition in the virtual routing table.

[0064] In order to quickly allocate the virtual nodes owned by a computing node to a new node when a data hotspot appears in a computing node, a hash mapping table data structure is used to save the virtual nodes owned by each computing node; due to the huge number of virtual nodes, in order to reduce the memory occupied by the mapping table, the mapping table only saves the minimum and maximum values ​​of the virtual node numbers owned by each computing node, while ensuring that the virtual nodes owned by each computing node are continuous.

[0065] For nodes that need hash partitioning, a virtual hash mapping table is designed. The key of the mapping table is the actual result partition object of the computing node, and the value of the mapping table is 0 to 2. 32-1. The virtual hash map is used to find out how many virtual hash values ​​a real result partition has. When initializing the map, the number of result partition objects is determined according to the parallelism of the downstream nodes, and then the ring is evenly divided into each result partition object according to the number of objects.

[0066] In order to quickly route data from virtual nodes to real physical computing nodes, the routing table marks the physical computing node to which each virtual node belongs. Also due to the number of virtual nodes, it is not suitable to design a one-to-one relationship between virtual nodes and computing nodes. This table will be designed as a data structure opposite to the mapping table - that is, the key-value pairs of the mapping table are interchanged. When searching for the corresponding virtual node, you only need to rewrite the key comparison method of the hash mapping table. If the current virtual node belongs to a certain key range, then the specific value can be determined, and the time complexity is low.

[0067] Design a virtual routing table for the nodes that need to be hash partitioned. The keys of the routing table are 0 to 2. 32 -1, the value is the real result partition object of the computing node. This routing table is used to find the real result partition object according to the virtual hash result after a data is hashed. Initialize the routing table according to the virtual mapping table.

[0068] Figure 2 A schematic diagram of dynamic node expansion of a task evaluation method based on dynamic expansion of Flink engine computing nodes in an embodiment of the present application.

[0069] like Figure 2 As shown in the figure, the client is responsible for submitting tasks. During the task, when the user observes a data hotspot on a node in the Flink web interface, he can send an instruction to increase the parallelism on the client. After receiving the instruction, the JobManager parses the instruction, changes the topology of the execution graph of the corresponding task, and sends instructions to the TaskManager to add result partitions, add computing nodes, and add connection channels. After receiving the instruction, the TaskManager adds result partitions, starts new computing nodes, adds input channels, and then changes the virtual hash map and virtual routing table.

[0070] Figure 3 This is a physical execution diagram of tasks in an embodiment of the present application.

[0071] like Figure 3As shown in the figure, at the beginning, each Source operator task receives user data, then generates two result partition objects according to the parallelism of the downstream Keyed target operator, and routes the data to the corresponding result partition objects according to the hash strategy; the Keyed operator task uses the corresponding input channel to obtain the user data processed by the upstream operator, and after processing the user data, it continues to output the data to its result partition. Finally, the Sink operator task receives the data processed by the Keyed task operator and outputs the user result.

[0072] Figure 4 This is a physical execution graph of tasks after node expansion of a task evaluation method based on dynamic expansion of Flink engine computing nodes in an embodiment of the present application.

[0073] like Figure 4 As shown in the figure, when a data hotspot appears in the target Keyed operator task, according to the instructions of the job manager, the upstream Source operator task changes the virtual hash mapping table and the virtual routing table, and adds a result partition; the target Keyed operator adds a computing node and its corresponding input channel and result partition; the downstream Sink operator adds an input channel, where the dotted box indicates the newly added nodes and data structures, and the newly added nodes and data structures do not affect the original data structures and nodes.

[0074] Figure 5 A circular diagram of a consistent hashing algorithm for a task evaluation method based on dynamic expansion of Flink engine computing nodes according to an embodiment of the present application.

[0075] like Figure 5 As shown, there is a circular address space, and each point on the ring is a number from 0 to 2. 32 Integer values ​​between -1, which represent the virtual partition hash values ​​of the compute node to connect to the downstream nodes.

[0076] Figure 6 Schematic diagram of the virtual hash mapping table change process of the task evaluation method based on the dynamic expansion of the Flink engine computing node according to an embodiment of the present application.

[0077] like Figure 6As shown, at the time of initialization, the virtual hash mapping table has three result partitions partition1, partition2, and partition3, and the corresponding virtual hash lists are [0,1,2,…,2^16], [2^16+1,…,2^24], and [2^24+1,…,2^32-1], respectively. When a data hotspot appears in the downstream node corresponding to partition3, the current node will create a real result partition partition4, and then divide the virtual hash value corresponding to partition3 into half to partition4. Finally, the virtual hash list corresponding to partition3 is [2^24+1,…,2^31], and the virtual hash list corresponding to partition4 is [2^31+1,…,2^32-1].

[0078] Figure 7 Schematic diagram of the virtual routing table change process of the task evaluation method based on the dynamic expansion of Flink engine computing nodes in an embodiment of the present application.

[0079] like Figure 7 As shown, the virtual routing table is changed according to the change of the virtual hash mapping table, specifically: all virtual hash values ​​corresponding to the newly created result partition partition4 are pointed to as result partition partition4 in the virtual routing table.

[0080] Figure 8 A structural diagram of a task evaluation device based on dynamic expansion of Flink engine computing nodes provided in Example 2 of the present application.

[0081] like Figure 8 As shown, the task evaluation device based on the dynamic expansion of the Flink engine computing node includes: a job manager, which is used to receive tasks sent by a client, and receive node expansion instructions sent by the client when a data hotspot appears, modify the topological structure of the execution graph of the task according to the node expansion instruction, and notify the upstream operator task of the target operator task to add a result partition and change the virtual hash mapping table and the virtual routing table, wherein the target operator task is the operator task where the data hotspot appears; notify the task manager to start a new computing node; notify the downstream operator task of the target operator task to add an input channel, wherein the input channel object added by the downstream operator task is connected to the result partition of the new computing node; process the user data according to the topological structure of the modified execution graph of the task through the task manager to obtain a task evaluation result;

[0082] The task manager is used to receive the instruction sent by the job manager, start the newly added computing node, initialize the input gateway and result partition of the newly added computing node, wherein the input gateway of the newly added computing node is connected to the result partition added by the upstream operator task; and is also used to process the user data according to the topological structure of the execution graph of the modified task to obtain the task evaluation result;

[0083] The client is used to build tasks, send tasks to the job manager, and send node expansion instructions to the job manager when data hotspots appear.

[0084] The task evaluation device based on the dynamic expansion of Flink engine computing nodes in the embodiment of the present application includes a job manager, which is used to receive tasks sent by a client, and receive node expansion instructions sent by the client when a data hotspot appears, modify the topological structure of the execution graph of the task according to the node expansion instruction, and notify the upstream operator task of the target operator task to add result partitions and change the virtual hash mapping table and the virtual routing table, wherein the target operator task is the operator task where the data hotspot appears; notify the task manager to start a new computing node; notify the downstream operator task of the target operator task to add an input channel, wherein the input channel object added by the downstream operator task is consistent with the newly added computing node The result partition of the point is connected; the task manager processes the user data according to the topological structure of the modified task execution graph to obtain the task evaluation result; the task manager is used to receive the instructions sent by the job manager, start the newly added computing node, initialize the input gateway and result partition of the newly added computing node, wherein the input gateway of the newly added computing node is connected to the result partition added by the upstream operator task; it is also used to process the user data according to the topological structure of the modified task execution graph to obtain the task evaluation result; the client is used to build the task, send the task to the job manager, and send the node expansion instruction to the job manager when the data hotspot appears. As a result, the computing nodes can be dynamically expanded when data hotspots appear, so as not to affect the correct operation of the system, and at the same time, the topological relationship between the original computing nodes is not affected, and the possibility of re-consuming data is avoided, which fundamentally solves the problem of data skew when the Flink engine processes data, improves the data calculation speed of the system, and meets the real-time scenario requirements of the task.

[0085] In order to implement the above embodiment, the present application also proposes a non-temporary computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the task evaluation method based on dynamic expansion of Flink engine computing nodes in the above embodiment is implemented.

[0086] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0087] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0088] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0089] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute the instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0090] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0091] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0092] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0093] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A task evaluation method based on dynamic expansion of Flink engine computing nodes, characterized in that: Applied to a job manager, the method comprises: Receiving a task sent by a client, wherein the task is generated by the client according to user data; Receive a node expansion instruction sent by the client when a data hotspot appears, modify the topological structure of the execution graph of the task according to the node expansion instruction, and notify the upstream operator task of the target operator task to add a result partition and change a virtual hash mapping table and a virtual routing table, wherein the target operator task is the operator task where the data hotspot appears; Start a newly added computing node through the task manager, initialize the input gateway and result partition of the newly added computing node, wherein the input gateway of the newly added computing node is connected to the result partition added by the upstream operator task; Notify the downstream operator task of the target operator task to add an input channel, wherein the added input channel of the downstream operator task is connected to the result partition of the newly added computing node; Processing the user data according to the modified topological structure of the execution graph of the task by the task manager to obtain a task evaluation result; The process of processing the user data according to the modified topological structure of the execution graph of the task by the task manager to obtain the task evaluation result includes: The task manager transmits the user data processed by the upstream operator task to the input gateway of the target operator task through the result partition, wherein the result partition added by the upstream operator task is connected to the input gateway of the newly added computing node; The user data received by the input gateway of the target operator task is divided into N original computing nodes and newly added computing nodes of the target operator task for processing, and the user data processed by the target operator task is transmitted to the input gateway of the downstream operator task through the result partition, wherein the result partition of the newly added computing node is connected to the input channel added by the downstream operator task; Distribute the user data received by the input gateway of the downstream operator task to the M computing nodes of the downstream operator task for processing, and transmit the user data processed by the downstream operator task to the next operator task for further processing; Wherein, N and M are positive integers.

2. The method according to claim 1, characterized in that The user data includes: one or more of the user's basic information, consumption data, loan data, income data, and social information.

3. The method according to claim 1, characterized in that The processing of the user data according to the modified topological structure of the execution graph of the task to obtain the task evaluation result also includes: Pulling user data through the initial operator task and transmitting it to the next operator task for processing according to the modified topological structure of the execution graph of the task; The processed user data is output through the output operator task to obtain the task evaluation result.

4. The method according to claim 1, characterized in that The changing of the virtual hash mapping table and the virtual routing table includes: By adding a mapping of a newly created result partition in the virtual hash mapping table through the upstream operator task, the virtual hash value corresponding to the result partition of the upstream operator task is evenly divided into half to the newly created result partition; According to the virtual hash mapping table, all virtual hash values ​​corresponding to the newly created result partition are mapped to the newly created result partition in the virtual routing table.

5. A task evaluation device based on dynamic expansion of Flink engine computing nodes, characterized in that: include: A job manager is used to receive tasks sent by a client, and receive node extension instructions sent by the client when a data hotspot appears, modify the topological structure of the execution graph of the task according to the node extension instruction, and notify the upstream operator task of the target operator task to add result partitions and change virtual hash mapping tables and virtual routing tables, wherein the target operator task is the operator task where the data hotspot appears; notify the task manager to start a new computing node; notify the downstream operator task of the target operator task to add an input channel, wherein the input channel object added by the downstream operator task is connected to the result partition of the new computing node; and process user data according to the modified topological structure of the execution graph of the task through the task manager to obtain a task evaluation result; The task manager is used to receive the instruction sent by the job manager, start the newly added computing node, initialize the input gateway and result partition of the newly added computing node, wherein the input gateway of the newly added computing node is connected to the result partition added by the upstream operator task; and is also used to process the user data according to the modified topological structure of the execution graph of the task to obtain the task evaluation result; The client is used to construct a task, send the task to the job manager, and send a node expansion instruction to the job manager when a data hotspot appears; The process of processing the user data according to the modified topological structure of the execution graph of the task by the task manager to obtain the task evaluation result includes: The task manager transmits the user data processed by the upstream operator task to the input gateway of the target operator task through the result partition, wherein the result partition added by the upstream operator task is connected to the input gateway of the newly added computing node; The user data received by the input gateway of the target operator task is divided into N original computing nodes and newly added computing nodes of the target operator task for processing, and the user data processed by the target operator task is transmitted to the input gateway of the downstream operator task through the result partition, wherein the result partition of the newly added computing node is connected to the input channel added by the downstream operator task; Distribute the user data received by the input gateway of the downstream operator task to the M computing nodes of the downstream operator task for processing, and transmit the user data processed by the downstream operator task to the next operator task for further processing; Wherein, N and M are positive integers.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images