Systems and methods for run parameter analysis and code redundancy optimization for stream processing systems

Abstract

The application provides a system and method for analyzing operation parameters and optimizing code redundancy of a stream processing system, comprising: software modules, a task construction module, an operator merging module and a stream processing system instance module; the stream processing system instance module is a stream processing system instance running on a server host, comprising a resource management and task allocation module, a work thread module, a parameter collection module, a strategy selection module and a code modification module; hardware modules, a stream processing system client host and a stream processing system server host. The application comprehensively uses static operator merging, dynamic query optimization and dynamic stream processing system redundant code optimization, realizes the combination of static optimization strategies and dynamic optimization strategies, and solves the problem of lacking static and dynamic combined optimization schemes.

Description

Technical Field

[0001] This invention relates to the field of computer big data processing technology, specifically to a system and method for analyzing operating parameters and optimizing code redundancy in stream processing systems. Background Technology

[0002] Big data is a product of the rapid development of information technology, representing a massive, rapidly growing, and diverse information asset. On the one hand, big data directly and accurately reflects user behavior, and the information it contains has significant guiding implications for the actions of developers, businesses, and decision-makers. On the other hand, due to the diversity of data, big data often contains a lot of redundant data with no practical value. Therefore, the core of big data processing lies in analyzing and processing the data, removing redundant data, and extracting valuable information from the massive amounts of data, thereby adding value to the data. Stream processing technology is one type of big data processing technology. It abstracts data into a continuously input data stream, performs efficient real-time analysis and processing of the data, and provides real-time feedback of the processing results to the user. The stream processing system is the computing system responsible for executing the process of receiving data, processing data, and outputting results. In scenarios where the real-time requirements of the data analysis process are high, stream processing systems play a crucial role.

[0003] Figure 1 This is a common architecture diagram of a stream processing system: A stream processing system consists of two parts: a client running locally on the user's computer and a server running on a dedicated server. The user submits computation task code to the stream processing system; this code includes the code for generating the data source and the logic for processing the data. The client analyzes the user-submitted task code through the task construction module, abstracting it into a directed acyclic graph (DAG) composed of different operators (e.g., ...). Figure 2Each operator is responsible for a portion of the computational logic in the task, and the result serves as the input for its downstream operators. Source operators represent the operators where the data source resides, and they have no upstream operators. Sink operators represent the operators to which the data processing results ultimately flow, and their results are output to external systems. After the operator graph is constructed, the client uploads it to the server. The server's resource management and task allocation modules then break down the entire computational task into different subtasks at the granularity of the operators in the operator graph, and allocate them to processes on different machines in the cluster. A process on a single machine can execute multiple subtasks simultaneously as worker threads. This module is also responsible for allocating computing resources (memory, CPU, etc.) to each machine and managing these resources uniformly. Each worker thread executing a subtask mainly consists of three modules: the input module is responsible for receiving data input. For the worker thread where the source operator is located, its input is the data generated by the data source. For the worker threads where other operators are located, its input is the output of the worker thread where the upstream operator is located. The data processing module is responsible for executing specific data processing tasks. The output module is responsible for outputting the processing results. For the worker thread where the sink operator is located, its results are output to the external system. For the worker threads where other operators are located, its results are output to the input module of the worker thread where the downstream operator is located.

[0004] The task code submitted by users is written from the perspective of users lacking knowledge of stream processing optimization. It may contain code that can be optimized (such as merging two adjacent computational logic code into one code). The above model directly translates the task code submitted by users into an operator graph, and then assigns the subtasks represented by each operator to worker threads on different machines. Due to the lack of optimization of the task code, the data communication overhead between different subtasks may be increased, thus preventing the stream processing system from achieving optimal data processing performance. During data processing, the input data will fluctuate over time, forming different data characteristics. The above processing model lacks the analysis and extraction of these dynamically changing data characteristics and cannot dynamically adjust the data processing strategy according to the data characteristics. The implementation of large-scale stream processing systems needs to consider many complex data processing scenarios. In order to ensure the robustness of the system, its implementation often uses more code abstraction and writes more general and safer implementation code. However, in specific data processing scenarios, this general code may affect the performance of data processing. The above processing model does not consider the dynamic optimization of the stream processing system implementation code.

[0005] Existing stream processing system optimization techniques either focus solely on static optimization before data processing (operator merging) or dynamic optimization during data processing (query optimization), lacking a combination of both. Furthermore, existing solutions lack analysis and abstraction of the overall data processing scenario characteristics; moreover, they do not optimize the stream processing system itself. These shortcomings prevent existing optimization techniques from addressing the problems of lacking a combination of static and dynamic optimization, lacking overall analysis and abstraction of data processing scenario characteristics, and lacking optimization of redundant code in stream processing systems. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the purpose of this invention is to provide a system and method for analyzing operating parameters and optimizing code redundancy in stream processing systems.

[0007] The system for analyzing operating parameters and optimizing code redundancy in a stream processing system according to the present invention includes:

[0008] Software modules: Task building module, operator merging module, and stream processing system instance module;

[0009] Hardware modules: Stream processing system client host and stream processing system server host;

[0010] The CPU in the client host of the stream processing system is responsible for performing task construction and operator merging, while the network card is responsible for sending the optimized operator graph to the server.

[0011] The stream processing system server hosts are divided into two categories: one type of host is the management host, whose CPU runs the stream processing system instance and is responsible for resource management and task allocation. The allocation results are sent to the computing host responsible for performing the computation via the network card; the other type of host is the computing host, on which the stream processing system instance is responsible for executing the specific computation logic. It obtains input data of subtasks on the same machine through memory and input data of subtasks between different machines through the network card. The CPU is responsible for executing the computation tasks and then outputting the computation results to memory or the network card.

[0012] The task construction module is responsible for receiving the computation task code submitted by the user and translating it into a directed acyclic operator graph.

[0013] The operator merging module performs optimization analysis on the operator graph, finds operators that can be merged, and merges them into a single operator;

[0014] The stream processing system instance module is a stream processing system instance running on the server host, including a resource management and task allocation module, a worker thread module, a parameter collection module, a strategy selection module, and a code modification module;

[0015] The resource management and task allocation module is responsible for allocating and managing the computing resources of each machine where the worker thread resides;

[0016] The working thread module receives input data, executes data processing logic, and outputs the data processing results;

[0017] The parameter collection module running on the computing host collects the parameters from the worker thread module, summarizes them, and sends them to the management host;

[0018] The strategy selection module running on the computing host analyzes the parameters and selects optimization strategies based on the analysis results, including algorithm optimization strategies for the worker thread module and code update strategies for the stream processing system instance module on the current host.

[0019] The parameter collection module running on the management host summarizes the processing parameters passed from each computing host, and also collects the execution parameters from the resource management and task allocation modules;

[0020] The policy selection module running on the management host analyzes the aggregated parameters and selects a code update policy for the stream processing system instance module on the management host based on the analysis results.

[0021] The code modification module runs simultaneously on the management host and the computing host, and dynamically modifies the code of the stream processing system instance module according to the update strategy.

[0022] Preferably, during operator merging, the computational tasks submitted by the user are abstracted into a StreamGraph using the Flink framework, and then the StreamGraph is optimized by operator merging to generate a JobGraph, which is then submitted to the server.

[0023] StreamGraph consists of StreamNodes and StreamEdges. Each StreamNode represents an operator, and each StreamEdge represents the data flow from one StreamNode to another.

[0024] A JobGraph consists of JobVertex and JobEdge. Each JobGraph represents one or more merged StreamNodes, and each JobEdge represents the data flow from one JobVertex to another.

[0025] Preferably, in the Java Virtual Machine of the computing host, performance parameters of all worker threads on the current computing host are monitored asynchronously by adding a background monitoring thread.

[0026] Preferably, a strategy mapping table is pre-built in the strategy selection module to map the monitored parameters to specific algorithm strategies; on the computing host, the strategy mapping table of the strategy selection module includes the algorithm strategy of the worker thread and the code update strategy for the current Flink instance.

[0027] Preferably, the Java assist tool is used in the code modification module to modify the code of the worker thread, adding or modifying Java code through dynamic compilation.

[0028] Preferably, the code modification module on the compute host is used to update the algorithm implementation code in the worker thread, and to update the code of the current Flink instance according to the strategy.

[0029] Preferably, the parameter collection module on the management host collects parameters from each computing host and execution parameters from the resource management and task allocation modules. The policy selection module selects the code update policy for the Flink instance from the policy mapping table based on the collected parameters. The code modification module updates the code of the current Flink instance based on the selected policy.

[0030] Preferably, worker threads on the compute host pass runtime parameters to the parameter collection module via Flink Metrics objects. The Metrics interface provided by Flink allows users to add custom performance parameter collection code to worker threads.

[0031] Preferably, the parameter collection module on the computing host formats the collected Flink Metrics objects according to a preset format, and finally obtains a parameter string.

[0032] According to the method for analyzing operating parameters and optimizing code redundancy in a stream processing system provided by this invention, the following steps are performed:

[0033] Step 1: Users write computational logic applications using the stream processing system programming API provided by the Flink client, and submit them to the Flink client after completion;

[0034] Step 2: The Flink client converts the computation logic into several StreamNodes, and generates StreamEdges between the StreamNodes based on the data dependencies between the computation logic in the application, ultimately generating a StreamGraph;

[0035] Step 3: The Flink client performs operator merging optimization on the StreamGraph, connecting mergeable StreamNodes into a JobVertex, generating JobEdges between JobVertexes based on data dependencies, and finally generating the JobGraph, which is then submitted to the Flink instance running on the management host.

[0036] Step 4: After receiving the JobGraph from the Flink client, the Flink instance running on the management host allocates computing resources and subtasks through the JobManager and submits the allocation results to the Flink instance running on the computing host.

[0037] Step 5: After receiving the allocation result from the management host, the Flink instance running on the compute host creates the corresponding Java worker thread and allocates computing resources to each thread. After the thread creation and resource allocation are completed, the Java worker thread begins to execute subtasks.

[0038] Step 6: The monitoring thread collects the running parameters of the Java thread and the running parameters of other worker threads transmitted from the computing host, and organizes them into parameter information in a preset format;

[0039] Step 7: Select the corresponding code update strategy from the strategy mapping table based on the parameter information in the preset format;

[0040] Step 8: Based on the selected code update strategy, use the Java assist tool to update the Flink instance code on the compute host / management host.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] (1) This invention integrates static operator merging, dynamic query optimization and dynamic stream processing system redundant code optimization, realizing the combination of static optimization strategy and dynamic optimization strategy, and solving the problem of lacking a dynamic-static combined optimization scheme.

[0043] (2) Collect all data processing parameters on the server host, abstract them comprehensively, and map them to the corresponding stream processing system code redundancy optimization strategy to solve the problem that the stream processing system lacks analysis and abstraction of the overall data processing scenario characteristics;

[0044] (3) Based on the optimization strategy obtained from the analysis, the implementation code of the stream processing system is dynamically modified to reduce functional redundancy and solve the problem that the existing optimization scheme lacks optimization of the redundant functions of the stream processing system itself. Attached Figure Description

[0045] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0046] Figure 1 This is an architecture diagram of a common stream processing system;

[0047] Figure 2 For operator graphs;

[0048] Figure 3 This is a system architecture diagram for existing technology 1;

[0049] Figure 4 Example of operator merging;

[0050] Figure 5 Here is a flowchart of the method in the prior art;

[0051] Figure 6 This is a system architecture diagram for existing technology 2;

[0052] Figure 7 Here is a flowchart of the method in the prior art 2;

[0053] Figure 8 This is a system architecture diagram of the present invention;

[0054] Figure 9 This describes the implementation of the invention in platform software and hardware;

[0055] Figure 10 In Flink Metrics format;

[0056] Figure 11 The formatted parameter format;

[0057] Figure 12 This is the format of the policy mapping table;

[0058] Figure 13 This is a flowchart of the core method of the present invention;

[0059] Figure 14 This is the input data format according to an embodiment of the present invention;

[0060] Figure 15 This is an operator diagram of an embodiment of the present invention;

[0061] Figure 16 This is a component structure diagram of an embodiment of the present invention;

[0062] Figure 17 This is a flowchart illustrating the implementation of a user-submitted task code according to an embodiment of the present invention.

[0063] Figure 18This is a flowchart illustrating the implementation process of constructing a StreamGraph according to an embodiment of the present invention;

[0064] Figure 19 This is a StreamGraph generated according to an embodiment of the present invention;

[0065] Figure 20 This is a flowchart illustrating the implementation of constructing a JobGraph according to an embodiment of the present invention;

[0066] Figure 21 This is a JobGraph generated according to an embodiment of the present invention;

[0067] Figure 22 A flowchart illustrating the allocation of computing resources and subtasks according to an embodiment of the present invention;

[0068] Figure 23 This is a flowchart illustrating the implementation of a Java thread executing a subtask according to an embodiment of the present invention.

[0069] Figure 24 This is a flowchart illustrating the implementation of parameter collection (computer host portion) according to an embodiment of the present invention;

[0070] Figure 25 This is a flowchart illustrating the implementation of an update strategy selection (computing host portion) according to an embodiment of the present invention.

[0071] Figure 26 This is a flowchart illustrating the implementation of the update code (computer host portion) according to an embodiment of the present invention;

[0072] Figure 27 This is a flowchart illustrating the implementation process of collecting parameters (management host portion) according to an embodiment of the present invention. Detailed Implementation

[0073] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0074] Example 1:

[0075] like Figure 3 This is a system architecture diagram of an existing operator merging scheme, which includes software modules: task building module, operator merging module, and stream processing system instance module. The stream processing system instance module includes resource management and task allocation module and worker thread module. The hardware modules are: stream processing system client host and stream processing system server host.

[0076] The functions of each module in the software are as follows: The task construction module is responsible for receiving the computation task code submitted by the user and translating it into a directed acyclic operator graph; the operator merging module is the core module of this technical solution. This module performs optimization analysis on the operator graph generated by the task construction module, finds operators that can be merged, and merges them into a single operator (such as...). Figure 4 The stream processing system instance module is a stream processing system instance running on the server host. The resource management and task allocation module is responsible for allocating and managing the computing resources of each machine where the worker threads reside. The worker thread module runs worker threads, which receive input data, execute data processing logic, and output data processing results.

[0077] The functions of each module in the hardware module are as follows: The CPU in the client host of the stream processing system is responsible for task construction and operator merging, and the network card is responsible for sending the optimized operator graph to the server. The server hosts of the stream processing system are divided into two categories: one type of host is the management host, whose CPU runs the stream processing system instance and is responsible for performing resource management and task allocation. The allocation results are sent to the server host responsible for performing the computation through the network card. The other type of host is the computing host, on which the stream processing system instance is responsible for executing the specific computation logic. It obtains input data through memory (subtasks on the same machine) or network card (subtasks between different machines), the CPU is responsible for executing the computation tasks, and then the computation results are output to memory, persistent storage, or network card.

[0078] The logical relationship between the hardware and software is as follows: The CPU of the client host is mainly responsible for running the task building and operator merging modules, and the network card is responsible for submitting the operator graph to the server host; the CPU of the management host is mainly used to run the resource management and task allocation modules on the stream processing system instance, and send the results to the computing host through the network card; each computing host can have multiple worker thread modules. The CPU on the computing host is used to run the worker threads in the stream processing system instance. The worker threads can obtain input data from the memory or network card of the computing host, and output the results to the memory, network card or persistent storage of the computing host.

[0079] like Figure 5 Here is a flowchart of the existing technology, with the specific steps as follows:

[0080] Step 1: User submits task. The user submits the written computation task code to the stream processing system client;

[0081] Step 2: Operator Graph Construction. The task construction module of the stream processing system client constructs the corresponding operator graph based on the task code;

[0082] Step 3: Operator merging. The operator merging module optimizes the operator graph, merges operators that meet the conditions, and then submits the optimized operator graph to the stream processing system server.

[0083] Step 4: Allocate computing resources and subtasks. The stream processing system instance on the server-side management host allocates computing resources to each computing host through the resource management and task allocation module. Simultaneously, it divides the system into subtasks based on the operator graph and then submits the resource and task allocation results to the computing hosts.

[0084] Step 5: Worker threads execute subtasks. The stream processing system instance on the compute host executes the subtasks as worker threads.

[0085] Existing technology one optimizes the operator graph submitted by the client to the server through operator merging. After merging, the number of operators in the operator graph is reduced, and data communication between merged operators can be performed in a single worker thread. This converts some inter-thread communication (subtasks on the same machine) or inter-process communication (subtasks on different machines) in the original operator graph into intra-thread communication, reducing data communication overhead during task execution to some extent. However, this technology still has some shortcomings: merging operators requires certain conditions to be met, and these conditions are often quite strict. Therefore, for operator graphs where none of the operators meet the conditions, the operator merging method will not have any optimization effect; operator merging optimization technology is a static optimization method, lacking analysis and abstraction of the characteristics of the data processing process. Once the operator graph is generated, it cannot be changed, thus failing to make dynamic adjustments during data processing to cope with the data characteristic fluctuations that may occur during big data processing; the focus of operator merging optimization technology is on the process of analyzing and generating the operator graph by the client of the stream processing system. For the functional redundancy problem in the aforementioned stream processing system code implementation, operator merging cannot play any role.

[0086] like Figure 6 This is a system architecture diagram of the existing technology 2 query optimization scheme, which includes software modules: a task construction module and a stream processing system instance module. The stream processing system instance module includes a resource management and task allocation module, a worker thread module, a parameter collection module, a strategy selection module, and a code modification module. The hardware modules are the stream processing system client host and the stream processing system server host.

[0087] The functions of each module in the software module are as follows: the functions of the task construction module, resource management and task allocation module, and worker thread module are consistent with the corresponding software modules in the aforementioned prior art 1; in the stream processing system instance module of the computing host, a parameter collection module, a strategy selection module, and a code modification module have been added. These three modules are the core modules in the query optimization scheme. During the data processing process, the data processing logic of the same subtask can be implemented with different specific algorithms. The parameter collection module is responsible for collecting some key performance parameters during the data processing process. The strategy selection module analyzes these parameters and then selects the optimal implementation algorithm based on the analysis results. The code modification module is responsible for updating the algorithm to the worker thread module.

[0088] The functions of each module in the hardware module are as follows: On the computing host, its CPU needs to perform additional parameter collection, strategy selection and code modification work, and the functions of the remaining hardware modules are the same as those of the hardware modules in the aforementioned prior art.

[0089] The logical relationship between the hardware and software is as follows: the CPU additional operating parameter collection module and the strategy selection module of the computing host, and the relationship between the remaining hardware and software is consistent with the hardware and software relationship in the aforementioned prior art.

[0090] like Figure 7 Here is a flowchart of the method in existing technology two, with the specific steps as follows:

[0091] Step 1: User submits task. The user submits the written computation task code to the stream processing system client;

[0092] Step 2: Operator Graph Construction. The task construction module of the stream processing system client constructs the corresponding operator graph based on the task code and submits the operator graph to the stream processing system server;

[0093] Step 3: Allocate computing resources and subtasks. The stream processing system instance module on the server-side management host allocates computing resources to each computing host through the resource management and task allocation module. At the same time, it divides subtasks according to the operator graph and then submits the resource and task allocation results to the computing hosts.

[0094] Step 4: Worker threads execute subtasks. The stream processing system instance module on the computing host executes subtasks as worker threads;

[0095] Step 5: Collect performance parameters. The parameter collection module asynchronously collects performance parameters from worker threads and passes them to the algorithm optimization module;

[0096] Step Six: Select the Optimal Algorithm. The strategy selection module selects the optimal data processing algorithm based on performance parameters;

[0097] Step 7: Update the algorithm. The code modification module updates the new algorithm in the worker thread module and returns to Step 4.

[0098] Existing technology two achieves dynamic optimization of the data processing process through query optimization. During data processing in a stream processing system, by analyzing the parameters collected by the parameter collection module, the strategy selection module can dynamically select the most suitable algorithm implementation for the current task. However, this technology still has some shortcomings: for subtasks with explicitly defined implementation logic by the user, the query optimization scheme cannot optimize them; each parameter collection module and strategy selection module runs independently on its respective computing host, limiting the performance parameters obtainable to the machine it resides on, lacking analysis and abstraction of the overall characteristics of the data processing scenario; similar to technology one, the query optimization scheme does not consider optimizing the stream processing system's own code, thus failing to effectively address the functional redundancy problem in the implementation of the stream processing system.

[0099] like Figure 8 This is a system architecture diagram of the present invention, which includes software modules: a task construction module, an operator merging module, and a stream processing system instance module. The stream processing system instance module includes a resource management and task allocation module, a worker thread module, a parameter collection module, a strategy selection module, and a code modification module. The hardware modules are a stream processing system client host and a stream processing system server host.

[0100] In the software modules, the functions of the task construction module, operator merging module, resource management and task allocation module, and worker thread module are consistent with the corresponding software modules in the aforementioned prior art 1 and prior art 2. The functions of the newly added software modules are as follows: The parameter collection module running on the computing host collects the parameters of each worker thread module on the machine, summarizes them, and sends them to the management host; the strategy selection module running on the computing host analyzes the parameters and selects an optimization strategy based on the analysis results. This optimization strategy can be an algorithm optimization strategy in the worker thread module or a code update strategy for the stream processing system instance module on the current host; the parameter collection module running on the management host summarizes the processing parameters transmitted from each computing host and also collects the execution parameters from the resource management and task allocation modules; the strategy selection module running on the management host analyzes the summarized parameters and selects a code update strategy for the stream processing system instance module on the management host based on the analysis results; the code modification module runs on both the management host and the computing host and dynamically modifies the code of the stream processing system instance module according to the update strategy.

[0101] In the hardware module: the network card of the computing host transmits the collected parameters to the management host, and its CPU needs to perform additional code modification tasks; the management host receives the parameters transmitted from the computing host through the network card, and its CPU needs to perform additional parameter collection, policy selection and code modification tasks. The functions of the remaining hardware modules are the same as the corresponding hardware modules in the aforementioned prior art 1 and prior art 2.

[0102] The logical relationship between the hardware and software is as follows: the network card of the computing host is additionally responsible for transmitting the running parameters collected by the parameter collection module; the CPU of the computing host additionally executes the code modification module; the network card of the management host additionally receives the running parameters of the worker threads collected by the computing host and passes them to the parameter collection module on the management host; the CPU of the management host additionally executes the parameter collection module, the policy selection module, and the code modification module. The remaining hardware and software relationships are consistent with those in the aforementioned prior art 1 and prior art 2.

[0103] The product implementation of this invention is the program code contained within the Flink stream processing system framework. Flink is an open-source stream processing framework developed by the Apache Software Foundation, whose core is a distributed stream data engine written in Java and Scala. Figure 9 As shown, in the Flink framework, both the management host and the compute hosts run Flink instances within the JVM. Resource management and task allocation on the management host are handled by the JobManager. Additionally, the JobManager periodically sends checkpoint signals to the compute hosts to persist computation progress, but checkpointing is not allowed. The Flink instances on the compute hosts primarily run Java threads, with each thread responsible for executing a subtask. Network communication between different hosts is implemented using the Netty network framework. This invention mainly consists of three optimization schemes for stream processing systems:

[0104] I. Operator Merging: The Flink framework itself has implemented operator merging, so this invention directly reuses Flink's operator merging mechanism. In Flink, the Flink client first abstracts the user-submitted computation task into a StreamGraph, then optimizes the StreamGraph by operator merging to generate a JobGraph, which is then submitted to the server. The StreamGraph mainly consists of StreamNodes and StreamEdges. Each StreamNode represents an operator, and each StreamEdge represents the data flow from one StreamNode to another. The JobGraph mainly consists of JobVertexes and JobEdges. Each JobGraph represents one or more merged StreamNodes, and each JobEdge represents the data flow from one JobVertex to another.

[0105] II. Query Optimization: In the JVM of the computing host, this invention asynchronously monitors the performance parameters of all worker threads on the current computing host by adding a background monitoring thread; a pre-built strategy mapping table is provided in the strategy selection module to map the monitored parameters to specific algorithm strategies; in the code modification module, this invention uses the Java assist tool to modify the code of worker threads. The Java assist tool is a class library used to process Java bytecode, which can add or modify Java code through dynamic compilation.

[0106] III. Stream Processing System Instance Code Optimization: On the compute host, the policy selection module's policy mapping table contains not only the algorithm policies for worker threads but also the code update policies for the current Flink instance. The code modification module on the compute host can update the algorithm implementation code in worker threads and also update the code of the current Flink instance according to the policies. The parameter collection module on the management host collects parameters from various compute hosts (via the network) and execution parameters from the resource management and task allocation modules. The policy selection module selects the code update policy for the Flink instance from the policy mapping table based on these parameters, and the code modification module updates the code of the current Flink instance according to the policies.

[0107] Worker threads on the compute host pass runtime parameters to the parameter collection module via Flink Metrics objects. The Flink Metrics interface allows users to add custom performance parameter collection code to worker threads. In the product implementation of this invention, the main structure of Flink Metrics is as follows: Figure 10Where: host id is a randomly generated 64-bit unsigned integer, and each server host has a unique host id; thread id is the id of the worker thread, which is a 64-bit unsigned integer; performance parameters are stored in a HashMap data structure, where the key is the parameter name and the value is the parameter value.

[0108] The parameter collection module on the compute host will collect Flink Metrics objects according to... Figure 11 The parameters are formatted according to a specific format, resulting in a parameter string, with different combinations of "host ID - thread ID" occupying one line. The parameter collection module on the management host collects parameters in the same format as... Figure 11 Consistent.

[0109] The format of the policy mapping table in the policy selection module is as follows: Figure 12 The parameter hash is the hash value of the aforementioned parameter string. The hash algorithm is SHA256. Each parameter hash corresponds to a code change strategy. In the change strategy, class represents the Java class that the strategy needs to modify, start line-end line represents the code of the class that the strategy needs to modify from the start line to the end line, and update code represents the implementation code of the new strategy.

[0110] Combination Figure 13 The core execution steps of the method of the present invention are described in detail below:

[0111] Step 1: User submits task code

[0112] Users write computational logic applications using the stream processing system programming API provided by the Flink client, and then submit them to the Flink client after completion.

[0113] Step 2: Construct StreamGraph

[0114] The Flink client converts the computational logic into several StreamNodes, and generates StreamEdges between StreamNodes based on the data dependencies between the computational logic in the application, ultimately generating a StreamGraph.

[0115] Step 3: Construct JobGraph

[0116] The Flink client performs operator merging optimization on the StreamGraph, connecting mergeable StreamNodes into a JobVertex, generating JobEdges between JobVertexes based on data dependencies, and finally generating the JobGraph, which is then submitted to the Flink instance running on the management host.

[0117] Step 4: Allocate computing resources and subtasks

[0118] After receiving the JobGraph from the Flink client, the Flink instance running on the management host allocates computing resources and subtasks through the JobManager and submits the allocation results to the Flink instance running on the compute host. In actual operation, there are usually multiple compute hosts.

[0119] Step 5: Java thread executes subtask

[0120] After receiving the allocation result from the management host, the Flink instance running on the compute host creates the corresponding Java worker thread and allocates computing resources to each thread. After the thread creation and resource allocation are completed, the Java worker thread begins to execute subtasks.

[0121] Step Six: Collect Parameters (Computing Host Section)

[0122] The monitoring thread collects the runtime parameters of the Java thread and organizes them into the format specified in this invention.

[0123] Step 7: Select Update Strategy (Compute Host Section)

[0124] Based on the formatted parameter information generated in step six, select the corresponding code update strategy from the strategy mapping table.

[0125] Step 8: Update the code (compute host portion)

[0126] Based on the code update strategy selected in step seven, use the Java assist tool to update the Flink instance code on the compute host.

[0127] Step 9: Collect Parameters (Management Host Section)

[0128] The monitoring thread collects the running parameters of worker threads transmitted from other computing hosts and organizes them into the format specified in this invention.

[0129] Step 10: Select Update Policy (Management Host Section)

[0130] Based on the formatted parameter information generated in step nine, select the corresponding code update strategy from the strategy mapping table.

[0131] Step 11: Update the code (management host section)

[0132] Based on the code update strategy selected in step 10, use the Java assist tool to update the Flink instance code on the management host.

[0133] Example 2:

[0134] The input data format of this invention is as follows: Figure 14 As shown, the input data consists of Bit objects, totaling 10M data entries. Each Bit object's `auction` and `price` properties are 64-bit unsigned integers. The operator graph is shown below. Figure 15 As shown, the source operator generates Bit objects through code and randomly fills the auction and price properties; the filter operator is responsible for filtering out data in the input data where auction equals 0; the window operator temporarily stores the Bit objects input every 10 seconds, groups the Bit objects according to the auction property value, and calculates the Bit object with the largest price value in each group of temporarily stored data after 10 seconds, and finally outputs the result to the collection operator; the collection operator outputs the result to the computer terminal.

[0135] Figure 16 This is the component structure diagram for Example 1. Java worker threads encapsulate parameters into Flink Metrics objects and pass them to the parameter collection module. The parameter collection module uses a custom parameter analysis and formatting program to format the Flink Metrics objects passed by each worker thread into the parameter format specified in this invention, and then passes them to the strategy selection module. In the strategy selection module, this example uses a Java HashMap data structure to store code update strategies. The key of the HashMap is the hash value of the formatted parameter string, and the value is the code update strategy organized according to the format specified in this invention. In the code modification module, this example uses the HotSwapper class provided by Java Assist to implement dynamic updates to the JVM code.

[0136] To implement runtime analysis and optimization functions in the stream processing system, the specific implementation steps in this embodiment are as follows:

[0137] Step 1: User submits task code

[0138] like Figure 17 Users write stream processing code using the Flink-provided stream processing system programming APIs (SourceFunction API for source operators, FilterFunction API for filter operators, keyby, TumblingProcessingWindow, and max APIs for window operators, and SinkFunction API for sink operators). Figure 15 The execution logic of each operator is then processed, and the completed application is submitted to the task building module of the Flink client.

[0139] Step 2: Construct StreamGraph

[0140] like Figure 18 Taking the application submitted by the user in step one as input, the Flink client constructs StreamNodes and StreamEdges for each operator, starting from the source operators and moving to the sink operators, based on the APIs used in the application. The final StreamGraph is as follows: Figure 19 .

[0141] Step 3: Construct JobGraph

[0142] like Figure 20 Taking the StreamGraph from step one as input, the Flink client checks for StreamNodes that can be merged and merges them into a JobVertex. In this embodiment, source operators and filter operators can be merged, and the resulting JobGraph is as follows. Figure 21 .

[0143] Step 4: Allocate computing resources and subtasks

[0144] like Figure 22 After receiving the JobGraph submitted by the client, the Flink server uses the JobManager to allocate computing resources and subtasks to each compute host and sends the allocation results to the compute host over the network.

[0145] Step 5: Java thread executes subtask

[0146] like Figure 23 After receiving the allocation result from the JobManager, the Flink instance on the compute host creates the corresponding Java worker thread based on the allocation result, allocates the corresponding computing resources to each thread, and then the worker thread starts to execute the computing task.

[0147] Step Six: Collect Parameters (Computing Host Section)

[0148] like Figure 24While the Java worker threads execute subtasks, the monitoring thread of the parameter collection module periodically collects Flink Metrics objects from all worker threads on the current host and formats these objects into the parameter format specified in this invention. In this embodiment, Flink Metrics mainly contains three parameters: isWindow, keys, and isCheckPoint. isWindow is a boolean value, and isWindow being true indicates that the current thread is executing a subtask of the window operator; keys represents the number of different keys (i.e., the number of different auctions) in the data processed by the current thread; isCheckPoint is a boolean value, and isCheckPoint being true indicates that the current thread needs to perform a checkpoint operation. Figure 24 In the implementation flowchart, the parameter collection module sequentially collects FlinkMetrics objects from the 1st to the nth worker thread (n is the number of Java worker threads); for each Flink Metrics object, the parameter collection module formats it as follows: Figure 11 The parameter collection module collects the parameter strings of each worker thread and passes them to the strategy selection module to select the update strategy for a specific worker thread. In addition, the parameter collection module also appends the parameter strings of each worker thread to a global string. When the module has collected the parameters of n worker threads, it will pass the global string to the strategy selection module to select the overall update strategy for the current Flink instance.

[0149] Step 7: Select Update Strategy (Compute Host Section)

[0150] like Figure 25The strategy selection module first calculates the hash value of the parameter string using the SHA256 algorithm, and then selects the corresponding update strategy from the strategy mapping table based on the hash value, passing it to the code modification module. In the implementation algorithm of the window operator, Flink's original implementation creates a trigger timer for each key value to output the calculation result. This implementation results in a large number of timer objects being created when the number of keys is large, thus affecting the data processing performance of the window operator. In this embodiment, when isWindow in the parameter string is true and keys > 1M, the strategy selection module selects a single-timer algorithm implemented in this invention to execute the window operator. This algorithm merges multiple timers in the original window into a single timer, thereby reducing the overhead of timers. Furthermore, when isCheckPoint in all parameter strings is false, the strategy selection modules on the compute host and management host update the code in the corresponding Flink instance, removing redundant code related to checkpoint operations.

[0151] Step 8: Update the code (compute host portion)

[0152] like Figure 26 Upon receiving the updated strategy from the strategy selection module, the code modification module first... Figure 12 The update strategy format is parsed to extract the Java class to be updated, the start line, the end line, and the update code. These parameters are then passed to the HotSwapper class of the Java assist, which performs the dynamic update of the Flink instance code.

[0153] Step 9: Collect Parameters (Management Host Section)

[0154] like Figure 27 The parameter collection module running on the management host receives parameter strings from other compute hosts, each string representing the overall operational status of the corresponding compute host. The parameter collection module on the management host appends these parameter strings to a global string. When the module has collected parameter strings from n compute hosts (n being the number of compute hosts), it passes the global string to the policy selection module to select the overall update policy for the Flink instances on the management host.

[0155] Step 10: Select Update Policy (Management Host Section)

[0156] The specific process is the same as step seven.

[0157] Step 11: Update the code (management host section)

[0158] The specific process is the same as step eight.

[0159] The results of this invention compared with those of the prior art are shown in the following table:

[0160] Table 1. Total time to execute the present invention using different optimization strategies.

[0161]

[0162] Table 1 shows that, using the stream processing system optimization strategy of the present invention, the total execution time is reduced by 24% compared to using operator merging alone, and by 19% compared to using query optimization alone.

[0163] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0164] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A system for analyzing operating parameters and optimizing code redundancy in stream processing systems, characterized in that, include: Software modules: Task building module, operator merging module, and stream processing system instance module; Hardware modules: Stream processing system client host and stream processing system server host; The CPU in the client host of the stream processing system is responsible for performing task construction and operator merging, while the network card is responsible for sending the optimized operator graph to the server. The stream processing system server hosts are divided into two categories: one type of host is the management host, whose CPU runs the stream processing system instance and is responsible for resource management and task allocation. The allocation results are sent to the computing host responsible for performing the computation via the network card; the other type of host is the computing host, on which the stream processing system instance is responsible for executing the specific computation logic. It obtains input data of subtasks on the same machine through memory and input data of subtasks between different machines through the network card. The CPU is responsible for executing the computation tasks and then outputting the computation results to memory or the network card. The task construction module is responsible for receiving the computation task code submitted by the user and translating it into a directed acyclic operator graph. The operator merging module performs optimization analysis on the operator graph, finds operators that can be merged, and merges them into a single operator; The stream processing system instance module is a stream processing system instance running on the server host, including a resource management and task allocation module, a worker thread module, a parameter collection module, a strategy selection module, and a code modification module; The resource management and task allocation module is responsible for allocating and managing the computing resources of each machine where the worker thread resides; The working thread module receives input data, executes data processing logic, and outputs the data processing results; The parameter collection module running on the computing host collects the parameters from the worker thread module, summarizes them, and sends them to the management host; The strategy selection module running on the computing host analyzes the parameters and selects optimization strategies based on the analysis results, including algorithm optimization strategies for the worker thread module and code update strategies for the stream processing system instance module on the current host. The parameter collection module running on the management host summarizes the processing parameters passed from each computing host, and also collects the execution parameters from the resource management and task allocation modules; The policy selection module running on the management host analyzes the aggregated parameters and selects a code update policy for the stream processing system instance module on the management host based on the analysis results. The code modification module runs simultaneously on the management host and the computing host, and dynamically modifies the code of the stream processing system instance module according to the update strategy.

2. The system for analyzing operating parameters and optimizing code redundancy in a stream processing system according to claim 1, characterized in that, During operator merging, the computation tasks submitted by users are abstracted into StreamGraphs using the Flink framework. Then, the StreamGraphs are optimized by operator merging to generate JobGraphs, which are then submitted to the server. StreamGraph consists of StreamNodes and StreamEdges. Each StreamNode represents an operator, and each StreamEdge represents the data flow from one StreamNode to another. A JobGraph consists of JobVertex and JobEdge. Each JobGraph represents one or more merged StreamNodes, and each JobEdge represents the data flow from one JobVertex to another.

3. The system for analyzing operating parameters and optimizing code redundancy in a stream processing system according to claim 1, characterized in that, In the Java Virtual Machine of the computing host, performance parameters of all worker threads on the current computing host can be monitored asynchronously by adding a background monitoring thread.

4. The system for analyzing operating parameters and optimizing code redundancy in a stream processing system according to claim 1, characterized in that, The strategy selection module pre-constructs a strategy mapping table to map the monitored parameters to specific algorithm strategies. On the compute host, the strategy mapping table of the strategy selection module includes the algorithm strategy of the worker thread and the code update strategy for the current Flink instance.

5. The system for analyzing operating parameters and optimizing code redundancy in a stream processing system according to claim 1, characterized in that, The Java assist tool is used in the code modification module to modify the code of worker threads, adding or modifying Java code through dynamic compilation.

6. The system for analyzing operating parameters and optimizing code redundancy in a stream processing system according to claim 1, characterized in that, The code modification module on the compute host is used to update the algorithm implementation code in worker threads, and to update the code of the current Flink instance according to the policy.

7. The system for analyzing operating parameters and optimizing code redundancy in a stream processing system according to claim 1, characterized in that, The parameter collection module on the management host collects parameters from various computing hosts and execution parameters from the resource management and task allocation modules. The policy selection module selects the code update policy for the Flink instance from the policy mapping table based on the collected parameters. The code modification module updates the code of the current Flink instance based on the selected policy.

8. The system for analyzing operating parameters and optimizing code redundancy in a stream processing system according to claim 1, characterized in that, Worker threads on the compute host pass runtime parameters to the parameter collection module via Flink Metrics objects. The Metrics interface provided by Flink allows users to add custom performance parameter collection code to worker threads.

9. The system for analyzing operating parameters and optimizing code redundancy in a stream processing system according to claim 1, characterized in that, The parameter collection module on the compute host formats the collected Flink Metrics objects according to a preset format, ultimately obtaining a parameter string.

10. A method for analyzing operating parameters and optimizing code redundancy in a stream processing system, characterized in that, Using the system for analyzing operating parameters and optimizing code redundancy of a stream processing system as described in any one of claims 1-9, the following steps are performed: Step 1: Users write computational logic applications using the stream processing system programming API provided by the Flink client, and submit them to the Flink client after completion; Step 2: The Flink client converts the computation logic into several StreamNodes, and generates StreamEdges between the StreamNodes based on the data dependencies between the computation logic in the application, ultimately generating a StreamGraph; Step 3: The Flink client performs operator merging optimization on the StreamGraph, connecting mergeable StreamNodes into a JobVertex, generating JobEdges between JobVertexes based on data dependencies, and finally generating the JobGraph, which is then submitted to the Flink instance running on the management host. Step 4: After receiving the JobGraph from the Flink client, the Flink instance running on the management host allocates computing resources and subtasks through the JobManager and submits the allocation results to the Flink instance running on the computing host. Step 5: After receiving the allocation result from the management host, the Flink instance running on the compute host creates the corresponding Java worker thread and allocates computing resources to each thread. After the thread creation and resource allocation are completed, the Java worker thread begins to execute subtasks. Step 6: The monitoring thread collects the running parameters of the Java thread and the running parameters of other worker threads transmitted from the computing host, and organizes them into parameter information in a preset format; Step 7: Select the corresponding code update strategy from the strategy mapping table based on the parameter information in the preset format; Step 8: Based on the selected code update strategy, use the Java assist tool to update the Flink instance code on the compute host / management host.

Images