Configurable Flink code implementation method and device, equipment, medium and product

Abstract

The invention provides a configurable Flink code implementation method and device, equipment, a medium and a product, and the method comprises the steps: integrating an execution flow chart and an operation chart corresponding to a to-be-configured Flink code into a topological graph; performing visualization processing on the topological graph, and sending the topological graph to a front-end interaction interface; in response to component configuration information for the topological graph acquired by the front-end interaction interface, adding null nodes to specified nodes and / or specified edges of the topological graph to obtain a configured topological graph, data processing logic of the null nodes corresponding to output logic included in the component configuration information; traversing the configured topological graph until all nodes of the topological graph are accessed, and outputting the configured Flink codes according to source codes corresponding to the accessed nodes. Through the embodiment of the invention, the visualization of the execution process of the Flink code is realized, the convenience and reliability of the configuration scheme of the Flink code are improved, and the maintenance cost of the Flink code is reduced.

Description

Background Art

[0002] At present, in order to meet people's increasingly high real-time data demands, using the Flink real-time computing engine for processing and processing is a good choice.

[0003] In related technologies, during the execution of Flink code, although the data flow of the task can be simply seen through the topology diagram, the data situation inside each operator cannot be directly observed. In addition, during the development, testing and exception debugging stages, programmers are usually required to embed logs in the code to record key information such as the task execution process and state changes, or write Flink code Metrics APIs in the code to monitor indicators such as data processing speed and processing delay to implement code verification logic and debug bugs. Moreover, after debugging, in order to avoid affecting the task performance of the code, these logs and monitoring codes often need to be deleted, which undoubtedly increases a lot of maintenance costs and time.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The purpose of the present disclosure is to provide a configurable Flink code implementation method, apparatus, device, medium and product, which are used to overcome the problems such as the complexity of the Flink code debugging process caused by the limitations and defects of related technologies, at least to a certain extent.

[0006] According to a first aspect of an embodiment of the present disclosure, a method for implementing configurable Flink code is provided, comprising: integrating an execution flow chart and a job graph corresponding to the Flink code to be configured into a topology graph; visualizing the topology graph and sending it to a front-end interactive interface; in response to component configuration information for the topology graph obtained by the front-end interactive interface, adding empty nodes to designated nodes and / or designated edges of the topology graph to obtain a configured topology graph, wherein the data processing logic of the empty nodes corresponds to the output logic included in the component configuration information; traversing the configured topology graph until it is determined that all nodes of the topology graph have been visited, and outputting the configured Flink code according to the source code corresponding to the visited nodes.

[0007] In an exemplary embodiment of the present disclosure, before integrating the execution flow chart and the job graph corresponding to the Flink code to be configured into a topology graph, it also includes:

[0008] Convert the Flink code to be configured into the flowchart through a third-party open source algorithm;

[0009] Annotating the Flink code in the flowchart to obtain the execution flowchart;

[0010] The Flink code to be configured is deployed on a server that supports the Flink stand-alone environment, and an image structure acquisition method inside the Flink stand-alone environment is called to obtain the job graph.

[0011] In an exemplary embodiment of the present disclosure, integrating the execution flow chart and the job graph corresponding to the Flink code to be configured into a topology graph includes:

[0012] After acquiring the execution flowchart and the job graph, determining the correspondence between the nodes of the job graph and the nodes of the execution flowchart based on the unique identifiers of the nodes;

[0013] The context information in the nodes of the execution flowchart is supplemented to the operation graph according to the corresponding relationship to obtain the topology graph.

[0014] In an exemplary embodiment of the present disclosure, in response to component configuration information for the topology map obtained by the front-end interactive interface, adding an empty node to a specified node and / or a specified edge of the topology map to obtain a configured topology map includes:

[0015] In response to the component configuration information for the topology graph obtained by the front-end interactive interface, the component configuration information is parsed to determine the attribute information of the output data of the configured Flink code, where the attribute information includes one of an output type, a link address, and an output logic, and the output logic includes at least one of a log printing logic, a log printing level, and a custom Metrics script logic;

[0016] In response to a drag operation on the empty node acquired by the front-end interactive interface, the empty node is added to a specified node and / or a specified edge of the topology graph to obtain a configured topology graph, wherein the configured topology graph includes annotated Flink code.

[0017] In an exemplary embodiment of the present disclosure, traversing the configured topology graph until it is determined that all nodes of the topology graph have been visited, and outputting the configured Flink code according to the source code corresponding to the visited nodes includes:

[0018] Traversing the configured topology map to determine whether all nodes of the topology map have been visited;

[0019] If it is determined that all nodes of the topology graph have been visited, the configured Flink code is output according to the source code corresponding to the visited nodes.

[0020] In an exemplary embodiment of the present disclosure, traversing the configured topology graph until it is determined that all nodes of the topology graph have been visited, and outputting the configured Flink code according to the source code corresponding to the visited nodes further includes:

[0021] Traversing the configured topology map to determine whether all nodes of the topology map have been visited;

[0022] If it is determined that all nodes of the topology graph have not been visited, then determining whether the unvisited nodes are test component nodes, and the test component nodes can execute the Flink code to be configured;

[0023] If it is determined that the unvisited node is not the test component node, the unvisited node is marked as visited, and the configured Flink code is output according to the source code corresponding to the visited node.

[0024] In an exemplary embodiment of the present disclosure, traversing the configured topology graph until it is determined that all nodes of the topology graph have been visited, and outputting the configured Flink code according to the source code corresponding to the visited nodes further includes:

[0025] Traversing the configured topology map to determine whether all nodes of the topology map have been visited;

[0026] If it is determined that all nodes of the topology graph have not been visited, then determining whether the unvisited nodes are test component nodes, and the test component nodes can execute the Flink code to be configured;

[0027] If it is determined that the unvisited node is the test component node, determining whether the test component node is the empty node;

[0028] If it is determined that the test component node is the empty node, insert a processFunction between the annotation of the source code of the upstream node of the empty node and the annotation of the source code of the downstream node of the empty node, the unique identifier of the processFunction is configured to include the identifier of the test component node and the name of the test component node, the processing logic of the processFunction corresponds to the output logic, and the processFunction is configured to use the output data of the upstream node as an input result and output the output result to the downstream node;

[0029] Mark the empty node inserted into the processFunction as visited, and read the source code corresponding to the visited node.

[0030] In an exemplary embodiment of the present disclosure, traversing the configured topology graph until it is determined that all nodes of the topology graph have been visited, and outputting the configured Flink code according to the source code corresponding to the visited nodes further includes:

[0031] Traversing the configured topology map to determine whether all nodes of the topology map have been visited;

[0032] If it is determined that all nodes of the topology graph have not been visited, then determining whether the unvisited nodes are test component nodes, and the test component nodes can execute the Flink code to be configured;

[0033] If it is determined that the unvisited node is the test component node, determining whether the test component node is the empty node;

[0034] If it is determined that the test component node is not the empty node, embedding a code block before the output method at the source code annotation position corresponding to the test component node, the code block being parsed and generated according to the output logic;

[0035] The test component nodes embedded in the code block are marked as visited, and then the source code corresponding to the visited nodes is determined.

[0036] In an exemplary embodiment of the present disclosure, it further includes:

[0037] Storing the topology map in a designated database;

[0038] Encapsulate at least one output result of the log, the flow measurement output, and the Metrics API collection information into a monitoring dashboard according to the link address of the test component node in the topology diagram;

[0039] The topology map is visualized and configured test component nodes are highlighted.

[0040] According to a second aspect of an embodiment of the present disclosure, a configurable Flink code implementation device is provided, including:

[0041] An integration module is configured to integrate the execution flow chart and the job graph corresponding to the Flink code to be configured into a topology graph;

[0042] A visualization module, configured to visualize the topology map and send it to a front-end interactive interface;

[0043] a configuration module, configured to add empty nodes to designated nodes and / or designated edges of the topology map in response to the component configuration information for the topology map obtained by the front-end interactive interface, so as to obtain a configured topology map, wherein the data processing logic of the empty nodes corresponds to the output logic included in the component configuration information;

[0044] The output module is configured to traverse the configured topology graph until it is determined that all nodes of the topology graph have been visited, and output the configured Flink code according to the source code corresponding to the visited nodes.

[0045] According to a third aspect of the present disclosure, there is provided an electronic device, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute any one of the methods described above based on instructions stored in the memory.

[0046] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a program is stored, and when the program is executed by a processor, the method for implementing the configurable Flink code as described in any one of the above items is implemented.

[0047] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program, characterized in that when the computer program is executed by a processor, the computer program implements the implementation method of the configurable Flink code as described in any one of the above items.

[0048] In the disclosed embodiment, the execution flow chart and the operation chart corresponding to the Flink code to be configured are integrated into a topology chart, the topology chart is visualized and sent to the front-end interactive interface, and in response to the component configuration information for the topology chart obtained by the front-end interactive interface, an empty node is added to the specified node and / or the specified edge of the topology chart to obtain the configured topology chart, and the data processing logic of the empty node corresponds to the output logic included in the component configuration information, that is, the user can configure log printing, Metrics monitoring, side stream output storage, etc., or configure the system built-in or custom module test method by clicking on a specific position of the topology chart on the front-end interactive interface, thereby realizing the convenient configuration of Flink code visualization. In addition, by traversing the configured topology chart until it is determined that all nodes of the topology chart have been visited, and outputting the configured Flink code according to the source code corresponding to the visited nodes, low-code, configurable operator-level data and performance monitoring are realized, and the decoupling of testing and development is realized, which saves the workload of users in writing code, avoids the probability of errors caused by adding and deleting code, and reduces the maintenance cost of Flink code.

[0049] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0051] Figure 1 A schematic diagram showing an exemplary system architecture of an implementation scheme of a configurable Flink code to which an embodiment of the present disclosure can be applied;

[0052] Figure 2 is a flowchart of a method for implementing a configurable Flink code in an exemplary embodiment of the present disclosure;

[0053] Figure 3 It is a schematic diagram of the architecture of another configurable Flink code implementation solution in an exemplary embodiment of the present disclosure;

[0054] Figure 4 It is a schematic diagram of the architecture of another configurable Flink code implementation solution in an exemplary embodiment of the present disclosure;

[0055] Figure 5 is a schematic diagram of a topology configuration module in an implementation scheme of a configurable Flink code in an exemplary embodiment of the present disclosure;

[0056] Figure 6 It is a flowchart of an implementation scheme of a configurable Flink code in an exemplary embodiment of the present disclosure;

[0057] Figure 7 It is a flowchart of another implementation scheme of configurable Flink code in an exemplary embodiment of the present disclosure;

[0058] Figure 8 It is a flowchart of another implementation scheme of configurable Flink code in an exemplary embodiment of the present disclosure;

[0059] Fig. 9 It is a flowchart of another implementation scheme of configurable Flink code in an exemplary embodiment of the present disclosure;

[0060] Fig.10 It is a flowchart of another implementation scheme of configurable Flink code in an exemplary embodiment of the present disclosure;

[0061] Fig.11It is a flowchart of another implementation scheme of configurable Flink code in an exemplary embodiment of the present disclosure;

[0062] Fig.12 It is a flowchart of another implementation scheme of configurable Flink code in an exemplary embodiment of the present disclosure;

[0063] Fig.13 It is a flowchart of another implementation scheme of configurable Flink code in an exemplary embodiment of the present disclosure;

[0064] Fig.14 It is a flowchart of another implementation scheme of configurable Flink code in an exemplary embodiment of the present disclosure;

[0065] Fig.15 It is a flowchart of another implementation scheme of configurable Flink code in an exemplary embodiment of the present disclosure;

[0066] Fig.16 is a block diagram of a configurable Flink code implementation device in an exemplary embodiment of the present disclosure;

[0067] Fig.17 is a block diagram of an electronic device in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0068] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0069] In addition, the accompanying drawings are only schematic diagrams of the present disclosure, and the same reference numerals in the drawings represent the same or similar parts, so their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities, which do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0070] The technical terms involved in the embodiments of the present disclosure include:

[0071] Flink: An open source streaming and batch processing framework designed to handle large-scale real-time and batch data streams. It has the characteristics of low latency and high throughput.

[0072] KeyBy: Group by key and split the data stream into different partitions according to the specified key.

[0073] Union: An operation that combines multiple data streams into one data stream.

[0074] Flink Metrics API: A tool interface for monitoring and measuring the performance of Flink applications. It allows users to collect and inspect various metrics of job execution, such as processing rate, latency, and memory usage.

[0075] Flink's JobGraph refers to the logical execution graph formed after the Flink code program is compiled and optimized. It describes the dependencies and data flows between the operators in the job, as well as the operation logic of each operator.

[0076] Figure 1 A schematic diagram showing an exemplary system architecture of an implementation scheme of configurable Flink code to which an embodiment of the present disclosure can be applied.

[0077] like Figure 1 As shown, the system architecture 100 may include one or more of terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0078] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 105 may be a server cluster composed of multiple servers.

[0079] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, portable computers, desktop computers, etc.

[0080] In some embodiments, the implementation method of the configurable Flink code provided in the embodiments of the present disclosure is generally executed by the server 105. Accordingly, the implementation device of the configurable Flink code is generally set in the terminal device 103 (which may also be the terminal device 101 or 102). In other embodiments, some terminals may have functions similar to those of the server device to execute the method.

[0081] The exemplary embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0082] Figure 2 It is a flowchart of a method for implementing a configurable Flink code in an exemplary embodiment of the present disclosure.

[0083] refer to Figure 2 , the implementation methods of configurable Flink code can include:

[0084] Step S202: Integrate the execution flow chart and the job graph corresponding to the Flink code to be configured into a topology graph.

[0085] In an exemplary embodiment of the present disclosure, the integrated topological graph is composed of a number of nodes and directed edges. Each node represents a Process, Filter, KeyBy and other processes, and the edge represents the upstream and downstream relationship between the nodes. In addition, the module will save a copy of the original code, and the copy is labeled. Each node and edge on the topological graph can correspond one-to-one with the label of the copy code.

[0086] Step S204: visualize the topology map and send it to the front-end interactive interface.

[0087] In an exemplary embodiment of the present disclosure, by visualizing the topology map and sending it to a front-end interactive interface, not only the visualization of the topology map is achieved, but also a front-end interface for configuring the topology map is provided to the user.

[0088] In an exemplary embodiment of the present disclosure, after the task of the Flink code is started, the user can click the configured test module in the topology map to observe the test data information of the monitoring configuration.

[0089] Step S206, in response to the component configuration information for the topology map obtained by the front-end interactive interface, add empty nodes to the specified nodes and / or specified edges of the topology map to obtain a configured topology map, and the data processing logic of the empty nodes corresponds to the output logic included in the component configuration information.

[0090] In an exemplary embodiment of the present disclosure, the configuration information includes log-related information, such as log printing logic, printing level, output location, etc., but is not limited thereto.

[0091] In an exemplary embodiment of the present disclosure, the configuration information also includes customizing some Metrics APIs to output operator data processing speed, data delay, etc., but is not limited thereto.

[0092] In an exemplary embodiment of the present disclosure, the configuration information also includes some operator-level data storage for monitoring and statistics of operator data status, but is not limited thereto.

[0093] In an exemplary embodiment of the present disclosure, flexible configuration of the topology diagram is achieved through the user's creation and / or dragging operations on empty nodes in the front-end interactive interface.

[0094] Step S208, traverse the configured topology graph until it is determined that all nodes of the topology graph have been visited, and output the configured Flink code according to the source code corresponding to the visited nodes.

[0095] In an exemplary embodiment of the present disclosure, the configured topology graph is traversed until it is determined that all nodes of the topology graph have been visited, and the configured Flink code is output according to the source code corresponding to the visited nodes, that is, the newly added test components in the topology graph are identified, reverse-parsed into separate Processes or several code blocks, embedded and combined with the corresponding tags of the code copy, and a new jar package is generated and finally output, thereby avoiding the error probability caused by adding and deleting code and reducing the maintenance cost of the Flink code.

[0096] Based on the embodiment of the present disclosure, by integrating the execution flow chart and the job graph corresponding to the Flink code to be configured into a topology map, the topology map is visualized and sent to the front-end interactive interface, in response to the component configuration information for the topology map obtained by the front-end interactive interface, an empty node is added to the specified node and / or the specified edge of the topology map to obtain the configured topology map, and the data processing logic of the empty node corresponds to the output logic included in the component configuration information, that is, the user can configure log printing, Metrics monitoring, side stream output storage, etc. by clicking on a specific position of the topology map on the front-end interactive interface, or configure the system built-in or custom module test method, thereby realizing a visualized and convenient configuration of the Flink code. In addition, by traversing the configured topology map until it is determined that all nodes of the topology map have been visited, and outputting the configured Flink code according to the source code corresponding to the visited nodes, low-code, configurable operator-level data and performance monitoring are realized, and decoupling of testing and development is realized, saving the workload of users in writing code, avoiding the probability of errors caused by adding and deleting code, and reducing the maintenance cost of Flink code.

[0097] The following is a detailed description of each step of the implementation method of the configurable Flink code.

[0098] In an exemplary embodiment of the present disclosure, before integrating the execution flow chart and the job graph corresponding to the Flink code to be configured into a topology graph, it also includes:

[0099] Convert the Flink code to be configured into the flowchart through a third-party open source algorithm;

[0100] Annotating the Flink code in the flowchart to obtain the execution flowchart;

[0101] The Flink code to be configured is deployed on a server that supports the Flink stand-alone environment, and an image structure acquisition method inside the Flink stand-alone environment is called to obtain the job graph.

[0102] In an exemplary embodiment of the present disclosure, after uploading the code address or the packaged Jar package to the topology analysis module, code conversion is first performed to generate an execution flow chart, and annotations are made on the source code.

[0103] In an exemplary embodiment of the present disclosure, these annotations correspond one-to-one to each node in the execution flow chart.

[0104] In an exemplary embodiment of the present disclosure, the operation diagram may be obtained by using third-party software such as Visustin, EasyStructure, etc., or may be implemented by self-developed software based on similar principles.

[0105] In an exemplary embodiment of the present disclosure, integrating the execution flow chart and the job graph corresponding to the Flink code to be configured into a topology graph includes:

[0106] After acquiring the execution flowchart and the job graph, determining the correspondence between the nodes of the job graph and the nodes of the execution flowchart based on the unique identifiers of the nodes;

[0107] The context information in the nodes of the execution flowchart is supplemented to the operation graph according to the corresponding relationship to obtain the topology graph.

[0108] In an exemplary embodiment of the present disclosure, the job graph (jobGraph) is used as the main graph, and the corresponding node of the execution flowchart is found through the unique identifier (UID) of each node, and then the node is added to the jobGraph according to the context information in the execution flowchart. Finally, these elements are combined into a new flowchart. In the above process, the annotation of the source code is also attached to the new flowchart.

[0109] In an exemplary embodiment of the present disclosure, in response to component configuration information for the topology map obtained by the front-end interactive interface, adding an empty node to a specified node and / or a specified edge of the topology map to obtain a configured topology map includes:

[0110] In response to component configuration information for the topology diagram obtained by the front-end interactive interface, the component configuration information is parsed to determine attribute information of output data of the configured Flink code, the attribute information including one of an output type, a link address, and an output logic, and the output logic including at least one of a log printing logic, a log printing level, and a custom Metrics script logic.

[0111] In an exemplary embodiment of the present disclosure, the output types include log printing, data storage, and custom metrics, but are not limited thereto.

[0112] In an exemplary embodiment of the present disclosure, the link address includes a log output location, a data storage type, a data storage link, etc., but is not limited thereto.

[0113] In an exemplary embodiment of the present disclosure, in response to a drag operation on the empty node obtained by the front-end interactive interface, the empty node is added to the specified node and / or specified edge of the topology graph to obtain a configured topology graph, and the configured topology graph includes annotated Flink code.

[0114] In an exemplary embodiment of the present disclosure, the configuration of the topology graph is implemented through empty nodes, especially the output of the specified node of the topology graph of the Flink code is configured, which improves the visual configuration of the Flink code and simplifies the configuration steps for obtaining the calculation results of the specified node.

[0115] In an exemplary embodiment of the present disclosure, traversing the configured topology graph until it is determined that all nodes of the topology graph have been visited, and outputting the configured Flink code according to the source code corresponding to the visited nodes includes:

[0116] Traversing the configured topology map to determine whether all nodes of the topology map have been visited;

[0117] If it is determined that all nodes of the topology graph have been visited, the configured Flink code is output according to the source code corresponding to the visited nodes.

[0118] In an exemplary embodiment of the present disclosure, traversing the configured topology graph until it is determined that all nodes of the topology graph have been visited, and outputting the configured Flink code according to the source code corresponding to the visited nodes further includes:

[0119] Traversing the configured topology map to determine whether all nodes of the topology map have been visited;

[0120] If it is determined that all nodes of the topology graph have not been visited, then determining whether the unvisited nodes are test component nodes, and the test component nodes can execute the Flink code to be configured;

[0121] If it is determined that the unvisited node is not the test component node, the unvisited node is marked as visited, and the configured Flink code is output according to the source code corresponding to the visited node.

[0122] In an exemplary embodiment of the present disclosure, traversing the configured topology graph until it is determined that all nodes of the topology graph have been visited, and outputting the configured Flink code according to the source code corresponding to the visited nodes further includes:

[0123] Traversing the configured topology map to determine whether all nodes of the topology map have been visited;

[0124] If it is determined that all nodes of the topology graph have not been visited, then determining whether the unvisited nodes are test component nodes, and the test component nodes can execute the Flink code to be configured;

[0125] If it is determined that the unvisited node is the test component node, determining whether the test component node is the empty node;

[0126] If it is determined that the test component node is the empty node, insert a processFunction between the annotation of the source code of the upstream node of the empty node and the annotation of the source code of the downstream node of the empty node, the unique identifier of the processFunction is configured to include the identifier of the test component node and the name of the test component node, the processing logic of the processFunction corresponds to the output logic, and the processFunction is configured to use the output data of the upstream node as an input result and output the output result to the downstream node;

[0127] Mark the empty node inserted into the processFunction as visited, and read the source code corresponding to the visited node.

[0128] In an exemplary embodiment of the present disclosure, if a node traversed to is a component node configured by an empty node, a new processFunction is inserted between the annotations of the source code between the upstream node and the downstream node, and its UID is set to the test component ID plus the test component name. The execution logic of processFunction is parsed and generated according to the configured output logic, and takes the output of the upstream node as input, and outputs the result to the downstream node.

[0129] In an exemplary embodiment of the present disclosure, traversing the configured topology graph until it is determined that all nodes of the topology graph have been visited, and outputting the configured Flink code according to the source code corresponding to the visited nodes further includes:

[0130] Traversing the configured topology map to determine whether all nodes of the topology map have been visited;

[0131] If it is determined that all nodes of the topology graph have not been visited, then determining whether the unvisited nodes are test component nodes, and the test component nodes can execute the Flink code to be configured;

[0132] If it is determined that the unvisited node is the test component node, determining whether the test component node is the empty node;

[0133] If it is determined that the test component node is not the empty node, embedding a code block before the output method at the source code annotation position corresponding to the test component node, the code block being parsed and generated according to the output logic;

[0134] The test component nodes embedded in the code block are marked as visited, and then the source code corresponding to the visited nodes is determined.

[0135] In an exemplary embodiment of the present disclosure, it further includes:

[0136] Storing the topology map in a designated database;

[0137] Encapsulate at least one output result of the log, the flow measurement output, and the Metrics API collection information into a monitoring dashboard according to the link address of the test component node in the topology diagram;

[0138] The topology map is visualized and configured test component nodes are highlighted.

[0139] Combine the following Figures 3 to 15 The implementation scheme of the configurable Flink code disclosed in the present invention is described in detail.

[0140] like Figure 3As shown, the steps for the developer 302 to configure the implementation architecture 300 based on the configurable Flink code include:

[0141] 1. After uploading the code to be tested or the jar package to the parsing topology module, the parsing topology module will parse the execution process of the Flink code and generate a flowchart. At the same time, the parsing topology module will also obtain the JobGraph data of the Flink code and integrate it with the flowchart to generate a new topology map. This topology map is composed of several nodes and directed edges. Each node represents a process such as Process, Filter, KeyBy, and the edge represents the upstream and downstream relationship between nodes. In addition, the parsing topology module will save a copy of the original code, which is annotated. Each node and edge on the topology map can correspond one-to-one with the annotation of the copy code.

[0142] 2. Further, the developer 302 can configure some built-in or customized test logic in the component configuration module, including log-related configurations, such as log printing logic, printing level, output location, etc., or customize some Metrics APIs to output operator data processing speed, data latency, etc., and also include some operator-level data storage for monitoring and statistics of operator data, etc., but not limited to this. These test logics will be stored in the database for use by the topology configuration module.

[0143] 3. Developer 302 can modify the topology map sent by the topology analysis module in the topology configuration module by dragging and linking the test components configured by the component configuration module on the nodes and edges of the topology map, and finally send the combined new topology map and the marked code copy to the topology reverse analysis module. The new topology map will also be sent to the monitoring dashboard module.

[0144] 4. The code parsing module obtains the new topology map and the marked code copy issued by the topology configuration module, identifies the newly added test components in the topology map, deparses them into separate processes or several code blocks, embeds and combines them with the corresponding tags of the code copy, generates a new jar package and outputs it.

[0145] 5. The monitoring dashboard module integrates and displays the topology map and corresponding test components (such as log address, database address, etc.) configured by the user. After the task is started, the user can click the configured test module in the topology map to observe and monitor the test data information of the configured code node.

[0146] exist Figure 3 On the basis of Figure 4 As shown, the hardware architecture 400 of the configurable Flink code of the embodiment of the present disclosure includes:

[0147] 1. The component configuration module includes two hardware modules, a component configurator and a component storage, between which operations such as query and storage can be performed. The developer 402 can enter the configuration information into the component configuration module in advance.

[0148] 2. If Figure 5 As shown, the topology parsing module includes a code parser and a stand-alone deployment environment. After the Jar package / code address is input into the topology parsing module, the job diagram and the flow chart can be obtained, and the topology diagram is obtained by integrating the two. The integrated topology diagram is input into the topology configuration module 500.

[0149] 3. The topology configuration module includes two parts: component acquisition and topology configurator. Component acquisition transmits the information configured by the developer 402 to the topology configurator.

[0150] 4. The code parsing module includes two parts: code parsing and code de-parsing. It obtains the new topology map and the marked code copy issued by the topology configuration module, identifies the newly added test components in the topology map, de-parses them into separate processes or several code blocks, embeds and combines them with the corresponding tags of the code copy, generates a new jar package / code address and outputs it.

[0151] 5. The topology configuration module includes two parts: topology storage and integrated dashboard. The monitoring dashboard module integrates and displays the user-configured topology map and corresponding test components (such as log address, database address, etc.). After the task is started, the user can click the configured test module in the topology map to observe and monitor the test data information of the configured code node.

[0152] In an exemplary embodiment of the present disclosure, the topology parsing module is responsible for implementing the conversion between the jar package source code and the user-configurable topology map. The specific process of the conversion is as follows:

[0153] After uploading the code address or the packaged Jar package to the topology analysis module, the data developer first performs code conversion to generate an execution flow chart and annotates the source code. These annotations correspond to each node in the execution flow chart.

[0154] In an exemplary embodiment of the present disclosure, the step of converting the code into an execution flowchart can be implemented with the help of software such as Visustin, EasyStructure, etc., or self-developed software based on similar principles.

[0155] In an exemplary embodiment of the present disclosure, three data sources are registered. Data source 1 and data source 2 are combined by Union and processed by Process, and then output to data source 3. The conversion and annotation execution process between the source code and the execution flowchart is as follows: Figure 6 As shown:

[0156] In an exemplary embodiment of the present disclosure, the correspondence between the source code and the annotations in the source code 602 includes:

[0157] Annotation 1: Register data source 1;

[0158] Note 2: Set uid "data source 1";

[0159] Annotation 3: Register data source 2;

[0160] Note 4: Set uid "data source 2";

[0161] Annotation 5: Register data source 3;

[0162] Note 6: Set uid "data source 3";

[0163] Annotation 7: Union data source 1 and data source 2;

[0164] Note 8: Key partition keyby;

[0165] Note 9: Processing function process;

[0166] Note 10: Set uid "process";

[0167] Callout 11: Output to data source 3.

[0168] At the same time, the source code will be deployed on the server of the Flink code stand-alone environment, and started and released by calling the getGraph method inside the Flink code. This step is to verify the correctness of the code and generate an execution flow chart 604 of the Flink code.

[0169] like Figure 7 As shown, JobGraph 704 is the main graph, and the execution flowchart is the auxiliary graph. Based on the annotated source code 702, a new flowchart is formed. The specific implementation method is as follows:

[0170] First, taking jobGraph 704 as the main graph, the corresponding execution flow graph node is found through the unique identifier (UID) of each node.

[0171] Secondly, if Figure 8 As shown, the context information in the execution flowchart 802 is supplemented to the jobGraph 804 .

[0172] Finally, these elements are combined into a new flowchart, such as Fig. 9 As shown, the source code annotation will also be attached to the combined flowchart 900.

[0173] The new flowchart is temporarily stored in the memory of the topology parsing module server in the form of a graph data structure, and then transmitted to the topology configuration module together with the annotated source code.

[0174] The component configuration module is responsible for centrally storing the test components required by Flink developers for configuration so that they can be used in the topology configuration module. The execution process is as follows:

[0175] Flink developers configure the output information required for testing, including output type, link information, and output logic. Output types include log printing, data storage, and custom metrics. Link information includes log output location, data storage type, and data storage link. Output logic includes log printing logic, log printing level, and uploaded custom metrics script logic.

[0176] In an exemplary embodiment of the present disclosure, the configuration parameters are shown in Table 1:

[0177] Table 1

[0178]

[0179] The output information configured by the user will be stored in the database of the component configuration module, supporting update and delete operations. Common database types include relational databases such as MySQL. At the same time, the component configuration module provides a query interface for the topology configuration module to use.

[0180] like Fig.11 As shown, according to the developer, in the topology configuration module 1106, the topology map delivered by the topology parsing module 1104 is combined with the test components configured by the component configuration module 1102 to generate a configured topology map including the test output information. The operation flow is as follows:

[0181] 1. If Fig.10 As shown, the topology data from the topology analysis module is graphically displayed, the output information configuration of the developer 1000 is obtained and stored, and an interface for corresponding configuration information is provided.

[0182] 2. If Fig.12 As shown, based on the original topology diagram 1200, the configurable interface displayed on the front end is as follows: Fig.13 As shown, the user can manually create zero or more empty nodes, such as empty node 1 and empty node 2, and connect these nodes to the original topology graph 1200 by dragging, connecting edges, etc. Fig.13 As shown, a new topology graph 1300 is formed.

[0183] 3. The user can manually click on any node or edge to configure it as any component in the component configuration module. This process requires querying the database of the component configuration module by ID.

[0184] 4. If Fig.14 As shown, the final topology diagram will be converted into a graph data structure 1400, and the added configuration contents include: "Log component 1, id: 768", "Log component 2, id: 772", "Custom metrics, id: 770", "Log component 2, id: 772" corresponds to "Storage component 1, id: 769", "Custom metrics, id: 770" corresponds to "Storage component 2, id: 771", and the graph data structure 1400 contains the annotated source code, which is passed to the code parsing module and the monitoring dashboard module.

[0185] like Fig.15 As shown in the figure, the code parsing module is responsible for re-parsing the topology graph passed by the component configuration module into new Flink code and outputting it. The specific method is to traverse the topology graph. The steps of a node access and processing include:

[0186] Step S1502, visit a node.

[0187] Step S1504, determine whether the node has been visited, if so, execute step S1506, if not, execute step S1508.

[0188] Step S1506, access the next node.

[0189] Step S1508, determine whether the node is marked as a test component node, if so, execute step S1510, if not, execute step S1526.

[0190] Step S1510, determine whether the node is configured as a null node, if so, execute step S1512, if not, execute step S1520.

[0191] Step S1512, parse the test component into processFunction.

[0192] Step S1514, set UID.

[0193] Step S1516, upstream and downstream docking input and output.

[0194] Step S1518, mark the node as "visited".

[0195] Step S1520, parsing the test component into code blocks.

[0196] Step S1522, embed the source code corresponding to the node before output.

[0197] Step S1524, mark the node as "visited".

[0198] Step S1526, mark the node as "visited".

[0199] The above node access and processing process can be summarized into the following key stages:

[0200] 1. Traverse the topology graph and mark the visited nodes as visited.

[0201] 2. For the test component node, process it according to the following two situations:

[0202] ① If the node is a component node configured by an empty node, a new processFunction is inserted between the source code annotations of the upstream and downstream nodes. Its UID is set to the test component ID plus the test component name. Its logic is parsed and generated according to the configured output logic, and the output of the upstream node is used as input to output the result to the downstream node.

[0203] ② If the node is a component node configured by other nodes, a code block is embedded before the output method of the source code annotation position corresponding to the node. The code block is parsed and generated according to the configured output logic. After processing, the node is marked as visited.

[0204] 3. Continue to traverse the topology graph until all nodes are marked as visited, and finally output new Flink code.

[0205] After receiving the topology diagram from the topology configuration module, the monitoring dashboard module performs the following steps:

[0206] 1. Store the topology map in the database for subsequent query.

[0207] 2. According to the link information of the test components in the topology diagram, the log log, flow measurement output storage, MetricsAPI and other storage are uniformly encapsulated into a monitoring dashboard.

[0208] 3. Visualize the topology map and highlight the configured test components.

[0209] In this way, when the data developer clicks on the highlighted part, the system will automatically jump to the corresponding monitoring dashboard page to facilitate the data developer to test and observe the data.

[0210] Corresponding to the above method embodiment, the present disclosure also provides a configurable Flink code implementation device, which can be used to execute the above method embodiment.

[0211] Fig.16 It is a block diagram of a configurable Flink code implementation device in an exemplary embodiment of the present disclosure.

[0212] refer to Fig.16 , the configurable Flink code implementation device 1600 may include:

[0213] An integration module 1602 is configured to integrate the execution flow chart and the job graph corresponding to the Flink code to be configured into a topology graph;

[0214] A visualization module 1604 is configured to visualize the topology map and send it to a front-end interactive interface;

[0215] A configuration module 1606 is configured to, in response to the component configuration information for the topology map obtained by the front-end interactive interface, add an empty node to a specified node and / or a specified edge of the topology map to obtain a configured topology map, wherein the data processing logic of the empty node corresponds to the output logic included in the component configuration information;

[0216] The output module 1608 is configured to traverse the configured topology graph until it is determined that all nodes of the topology graph have been visited, and output the configured Flink code according to the source code corresponding to the visited nodes.

[0217] In an exemplary embodiment of the present disclosure, the integration module 1602 is further configured to:

[0218] Convert the Flink code to be configured into the flowchart through a third-party open source algorithm;

[0219] Annotating the Flink code in the flowchart to obtain the execution flowchart;

[0220] The Flink code to be configured is deployed on a server that supports the Flink stand-alone environment, and an image structure acquisition method inside the Flink stand-alone environment is called to obtain the job graph.

[0221] In an exemplary embodiment of the present disclosure, the integration module 1602 is further configured to:

[0222] After acquiring the execution flowchart and the job graph, determining the correspondence between the nodes of the job graph and the nodes of the execution flowchart based on the unique identifiers of the nodes;

[0223] The context information in the nodes of the execution flowchart is supplemented to the operation graph according to the corresponding relationship to obtain the topology graph.

[0224] In an exemplary embodiment of the present disclosure, the configuration module 1606 is further configured to:

[0225] In response to the component configuration information for the topology graph obtained by the front-end interactive interface, the component configuration information is parsed to determine the attribute information of the output data of the configured Flink code, where the attribute information includes one of an output type, a link address, and an output logic, and the output logic includes at least one of a log printing logic, a log printing level, and a custom Metrics script logic;

[0226] In response to a drag operation on the empty node acquired by the front-end interactive interface, the empty node is added to a specified node and / or a specified edge of the topology graph to obtain a configured topology graph, wherein the configured topology graph includes annotated Flink code.

[0227] In an exemplary embodiment of the present disclosure, the output module 1608 is further configured to:

[0228] Traversing the configured topology map to determine whether all nodes of the topology map have been visited;

[0229] If it is determined that all nodes of the topology graph have been visited, the configured Flink code is output according to the source code corresponding to the visited nodes.

[0230] In an exemplary embodiment of the present disclosure, the output module 1608 is further configured to:

[0231] Traversing the configured topology map to determine whether all nodes of the topology map have been visited;

[0232] If it is determined that all nodes of the topology graph have not been visited, then determining whether the unvisited nodes are test component nodes, and the test component nodes can execute the Flink code to be configured;

[0233] If it is determined that the unvisited node is not the test component node, the unvisited node is marked as visited, and the configured Flink code is output according to the source code corresponding to the visited node.

[0234] In an exemplary embodiment of the present disclosure, the output module 1608 is further configured to:

[0235] Traversing the configured topology map to determine whether all nodes of the topology map have been visited;

[0236] If it is determined that all nodes of the topology graph have not been visited, then determining whether the unvisited nodes are test component nodes, and the test component nodes can execute the Flink code to be configured;

[0237] If it is determined that the unvisited node is the test component node, determining whether the test component node is the empty node;

[0238] If it is determined that the test component node is the empty node, insert a processFunction between the annotation of the source code of the upstream node of the empty node and the annotation of the source code of the downstream node of the empty node, the unique identifier of the processFunction is configured to include the identifier of the test component node and the name of the test component node, the processing logic of the processFunction corresponds to the output logic, and the processFunction is configured to use the output data of the upstream node as an input result and output the output result to the downstream node;

[0239] Mark the empty node inserted into the processFunction as visited, and read the source code corresponding to the visited node.

[0240] In an exemplary embodiment of the present disclosure, the output module 1608 is further configured to:

[0241] Traversing the configured topology map to determine whether all nodes of the topology map have been visited;

[0242] If it is determined that all nodes of the topology graph have not been visited, then determining whether the unvisited nodes are test component nodes, and the test component nodes can execute the Flink code to be configured;

[0243] If it is determined that the unvisited node is the test component node, determining whether the test component node is the empty node;

[0244] If it is determined that the test component node is not the empty node, embedding a code block before the output method at the source code annotation position corresponding to the test component node, the code block being parsed and generated according to the output logic;

[0245] The test component nodes embedded in the code block are marked as visited, and then the source code corresponding to the visited nodes is determined.

[0246] In an exemplary embodiment of the present disclosure, the configurable Flink code implementation device 1600 is further used to:

[0247] Storing the topology map in a designated database;

[0248] Encapsulate at least one output result of the log, the flow measurement output, and the Metrics API collection information into a monitoring dashboard according to the link address of the test component node in the topology diagram;

[0249] The topology map is visualized and configured test component nodes are highlighted.

[0250] Since the functions of the device 1600 have been described in detail in the corresponding method embodiments, the present disclosure will not elaborate on them here.

[0251] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0252] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0253] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as circuits, modules or systems.

[0254] Refer to the following Fig.17 1700 according to this embodiment of the present disclosure is described. Fig.17 The electronic device 1700 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0255] like Fig.17 As shown, the electronic device 1700 is in the form of a general computing device. The components of the electronic device 1700 may include but are not limited to: at least one processing unit 1710, at least one storage unit 1720, and a bus 1730 connecting different system components (including the storage unit 1720 and the processing unit 1710).

[0256] The storage unit stores program codes, which can be executed by the processing unit 1710, so that the processing unit 1710 performs the steps according to various exemplary embodiments of the present disclosure described in the above exemplary method section of this specification. For example, the processing unit 1710 can perform the method shown in the embodiment of the present disclosure.

[0257] The storage unit 1720 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 17201 and / or a cache storage unit 17202 , and may further include a read-only storage unit (ROM) 17203 .

[0258] The storage unit 1720 may also include a program / utility 17204 having a set (at least one) of program modules 17205, such program modules 17205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0259] Bus 1730 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0260] The electronic device 1700 may also communicate with one or more external devices 1740 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1700, and / or communicate with any device that enables the electronic device 1700 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1750. Furthermore, the electronic device 1700 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 1760. As shown, the network adapter 1760 communicates with other modules of the electronic device 1700 via a bus 1730. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0261] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0262] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to perform the steps according to various exemplary implementations of the present disclosure described in the above exemplary method section of the present specification.

[0263] The program product for implementing the above method according to the embodiment of the present disclosure can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0264] Among them, the readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0265] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0266] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0267] In an exemplary embodiment of the present disclosure, a computer program product is also provided, which can be loaded or stored in any combination of one or more readable media, and can be written in any combination of one or more programming languages ​​to perform the program code of the present disclosure, including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as C language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or completely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0268] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0269] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are to be considered exemplary only, and the true scope and concept of the present disclosure are indicated by the claims.

Claims

1. A method for implementing a configurable Flink code, characterized in that: include: Integrate the execution flow chart and job graph corresponding to the Flink code to be configured into a topology graph; Visualizing the topology map and sending it to a front-end interactive interface; In response to the component configuration information for the topology map obtained by the front-end interactive interface, an empty node is added to a specified node and / or a specified edge of the topology map to obtain a configured topology map, wherein the data processing logic of the empty node corresponds to the output logic included in the component configuration information; The configured topology graph is traversed until it is determined that all nodes of the topology graph have been visited, and the configured Flink code is output according to the source code corresponding to the visited nodes.

2. The method for implementing the configurable Flink code according to claim 1, characterized in that: Before integrating the execution flow chart and job graph corresponding to the Flink code to be configured into a topology graph, it also includes: Convert the Flink code to be configured into the flowchart through a third-party open source algorithm; Annotating the Flink code in the flowchart to obtain the execution flowchart; The Flink code to be configured is deployed on a server that supports the Flink stand-alone environment, and an image structure acquisition method inside the Flink stand-alone environment is called to obtain the job graph.

3. The method for implementing the configurable Flink code according to claim 2, characterized in that: Integrate the execution flow chart and job graph corresponding to the Flink code to be configured into a topology graph including: After acquiring the execution flowchart and the job graph, determining the correspondence between the nodes of the job graph and the nodes of the execution flowchart based on the unique identifiers of the nodes; The context information in the nodes of the execution flowchart is supplemented to the operation graph according to the corresponding relationship to obtain the topology graph.

4. The method for implementing the configurable Flink code according to claim 2, characterized in that: In response to the component configuration information for the topology map obtained by the front-end interactive interface, adding an empty node to a specified node and / or a specified edge of the topology map to obtain a configured topology map includes: In response to the component configuration information for the topology graph obtained by the front-end interactive interface, the component configuration information is parsed to determine the attribute information of the output data of the configured Flink code, where the attribute information includes one of an output type, a link address, and an output logic, and the output logic includes at least one of a log printing logic, a log printing level, and a custom Metrics script logic; In response to a drag operation on the empty node acquired by the front-end interactive interface, the empty node is added to a specified node and / or a specified edge of the topology graph to obtain a configured topology graph, wherein the configured topology graph includes annotated Flink code.

5. The method for implementing the configurable Flink code according to claim 4, characterized in that: Traversing the configured topology until all nodes of the topology are determined to have been visited, and outputting the configured Flink code according to the source code corresponding to the visited nodes includes: Traversing the configured topology map to determine whether all nodes of the topology map have been visited; If it is determined that all nodes of the topology graph have been visited, the configured Flink code is output according to the source code corresponding to the visited nodes.

6. The method for implementing the configurable Flink code according to claim 4, characterized in that: Traversing the configured topology graph until all nodes of the topology graph are determined to have been visited, and outputting the configured Flink code according to the source code corresponding to the visited nodes also includes: Traversing the configured topology map to determine whether all nodes of the topology map have been visited; If it is determined that all nodes of the topology graph have not been visited, then determining whether the unvisited nodes are test component nodes, and the test component nodes can execute the Flink code to be configured; If it is determined that the unvisited node is not the test component node, the unvisited node is marked as visited, and the configured Flink code is output according to the source code corresponding to the visited node.

7. The method for implementing the configurable Flink code according to claim 4, characterized in that: Traversing the configured topology graph until all nodes of the topology graph are determined to have been visited, and outputting the configured Flink code according to the source code corresponding to the visited nodes also includes: Traversing the configured topology map to determine whether all nodes of the topology map have been visited; If it is determined that all nodes of the topology graph have not been visited, then determining whether the unvisited nodes are test component nodes, and the test component nodes can execute the Flink code to be configured; If it is determined that the unvisited node is the test component node, determining whether the test component node is the empty node; If it is determined that the test component node is the empty node, insert a processFunction between the annotation of the source code of the upstream node of the empty node and the annotation of the source code of the downstream node of the empty node, the unique identifier of the processFunction is configured to include the identifier of the test component node and the name of the test component node, the processing logic of the processFunction corresponds to the output logic, and the processFunction is configured to use the output data of the upstream node as an input result and output the output result to the downstream node; Mark the empty node inserted into the processFunction as visited, and read the source code corresponding to the visited node.

8. The method for implementing the configurable Flink code according to claim 4, characterized in that: Traversing the configured topology graph until all nodes of the topology graph are determined to have been visited, and outputting the configured Flink code according to the source code corresponding to the visited nodes also includes: Traversing the configured topology map to determine whether all nodes of the topology map have been visited; If it is determined that all nodes of the topology graph have not been visited, then determining whether the unvisited nodes are test component nodes, and the test component nodes can execute the Flink code to be configured; If it is determined that the unvisited node is the test component node, determining whether the test component node is the empty node; If it is determined that the test component node is not the empty node, embedding a code block before the output method at the source code annotation position corresponding to the test component node, the code block being parsed and generated according to the output logic; The test component nodes embedded in the code block are marked as visited, and then the source code corresponding to the visited nodes is determined.

9. The method for implementing configurable Flink code according to claim 4, characterized in that: Also includes: Storing the topology map in a designated database; Encapsulate at least one output result of the log, the flow measurement output, and the Metrics API collection information into a monitoring dashboard according to the link address of the test component node in the topology diagram; The topology map is visualized and configured test component nodes are highlighted.

10. A configurable Flink code implementation device, characterized in that: include: An integration module is configured to integrate the execution flow chart and the job graph corresponding to the Flink code to be configured into a topology graph; A visualization module, configured to visualize the topology map and send it to a front-end interactive interface; a configuration module, configured to add empty nodes to designated nodes and / or designated edges of the topology map in response to the component configuration information for the topology map obtained by the front-end interactive interface, so as to obtain a configured topology map, wherein the data processing logic of the empty nodes corresponds to the output logic included in the component configuration information; The output module is configured to traverse the configured topology graph until it is determined that all nodes of the topology graph have been visited, and output the configured Flink code according to the source code corresponding to the visited nodes.

11. An electronic device, characterized in that: include: Memory; as well as A processor coupled to the memory, the processor being configured to execute the implementation method of the configurable Flink code according to any one of claims 1 to 9 based on instructions stored in the memory.

12. A computer-readable storage medium having a program stored thereon, wherein when the program is executed by a processor, the method for implementing the configurable Flink code according to any one of claims 1 to 9 is implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for implementing the configurable Flink code as described in any one of claims 1 to 9 is implemented.

Images