Dynamic calculation topology optimization method based on Flink SQL (Structured Query Language)

Abstract

The invention discloses a dynamic calculation topology optimization method based on an Flink SQL, relates to the field of big data and real-time calculation, and aims to re-realize self-defined calculation topology arrangement and complete dynamic compression of different SQL calculation topologies by matching a native Flink SQL execution plan. The invention discloses an Flink SQL (Structured Query Language) engine-based dynamic combination, adjustment and removal method, and aims at constructing an innovative optimization method based on an Flink SQL engine, and realizing dynamic combination, adjustment and removal of multiple types of SQL calculation operators including Project, Filter, Aggregate, Window and the like in a unified calculation topology aiming at typical calculation modes including data extraction, window aggregation and the like which are covered by a streaming SQL calculation scene based on the same data source, so as to realize dynamic combination, adjustment and removal of multiple types of SQL calculation operators including Project, Filter, Aggregate, Window and the like. Sharing and multiplexing of computing resources are achieved through an intelligent resource scheduling mechanism, dynamic evolution of real-time SQL logic is effectively supported, and the computing efficiency and the resource utilization rate are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of big data and real-time computing, and in particular relates to a dynamic computing topology optimization method based on Flink SQL. Background Technology

[0002] Flink SQL, a high-performance streaming computing framework, is capable of efficiently computing massive amounts of real-time data. One of its core features is the use of Operator Chain technology to optimize and merge multiple SQL tasks submitted by the client at once during the computing topology construction phase. By chaining the operators of multiple SQL tasks together to form a single computing topology that shares computing resources, resource sharing among multiple SQL tasks and efficient utilization of computing resources are effectively achieved.

[0003] In real-world business scenarios, there is a typical requirement for data extraction and computation from the same source: multiple SQL tasks need to perform calculations on the same data source, i.e., the Source table. To ensure the continuity of streaming computation, the business side needs to dynamically add SQL tasks or modify existing SQL logic during the execution of computation tasks. However, current technologies cannot support such dynamic adjustments.

[0004] Currently, the Operator Chain optimization strategy implemented in the Flink SQL computation engine generates a static computation topology, which cannot dynamically update the computation topology and computational logic tasks. If a user needs to add SQL computations to an already running Job, they must stop the currently running Job, modify the SQL, and then resubmit the Job. The main shortcomings of the current Flink SQL computation optimization include: 1. Lack of dynamic topology update capability: The current Flink SQL computing framework does not support dynamic updates to the computing topology during runtime. The topology of all SQL tasks is fixed upon job submission and cannot be flexibly adjusted during operation.

[0005] 2. Risk of computational task interruption and data loss due to dynamic adjustments: To add or modify SQL tasks, the existing computing topology must be stopped and a new computing topology must be generated. This process can lead to the following problems: Computation task interruption: The execution of the existing job is forcibly terminated, affecting the normal execution of other SQL tasks in the same source data; Data processing delay: Data needs to be reprocessed after the job is restarted, which may cause delays in business data. Data loss risk: If the data source cannot replay the data (e.g., real-time data stream has no historical records), the data during the period when the topology is not rerun will not be recoverable, resulting in the loss of business data. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a dynamic computation topology optimization method based on Flink SQL to address the shortcomings of the prior art.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: A dynamic computation topology optimization method based on Flink SQL is proposed. By matching the native Flink SQL execution plan, a custom computation topology orchestration is reimplemented to achieve dynamic compression of different SQL computation topologies. The method includes the following steps. Step 1: Perform SQL rule matching to determine whether the current SQL meets the topology merging conditions; Step 2: Perform SQL parsing, extract SQL operators, reconstruct the computation object, and serialize it; Step 3: The computation object is sent to the worker node TaskManager. The worker node TaskManager receives the computation object, compiles the object, compiles the computation logic using dynamic code compilation technology, and adds the current computation operator to the computation list. Step 4: After the data calculation is completed, it is sent to the computing unit for projection and final output.

[0008] As a further preferred embodiment of the dynamic computation topology optimization method based on Flink SQL of the present invention, in the steps, task template ID matching is performed by hint matching, and extraction SQL with the same source and aggregation SQL with the same aggregation conditions are matched by same source SQL; if the current execution plan is an execution plan of a template type defined by the system, and is an extraction task or an aggregation task, then the computation operator merging condition is met.

[0009] As a further preferred embodiment of the dynamic computation topology optimization method based on Flink SQL of the present invention, in step 2, the SQL operator includes Project, Filter, Aggregate, and Window. It is determined whether a computation topology with the same source is already running; if not, a new computation topology is started; if a computation topology with the same source already exists, subsequent computation dynamic update operations are performed. If computation dynamic update is required, the relevant SQL operators are first processed to generate code descriptions related to these operators, and these code snippets are encapsulated into a Calculate object. At the same time, the object is serialized and sent to the specified worker node TaskManager.

[0010] As a further preferred embodiment of the dynamic computation topology optimization method based on Flink SQL of the present invention, in step 3, the worker node TaskManager receives the Calculate object that needs to be updated, performs a serialization operation on the Calculate object that needs to be updated; parses the Calculate object that needs to be updated and extracts the operator descriptions therein, including Project, Filter, Aggregate, and Window; and regenerates the code of the relevant computation object using the code snippets of the relevant operators. The predicate code template is PredicateCalc; The code template for projection classes is ProjectCalc; The code template for aggregation classes is AggregateCalc; The generated code is compiled, and the computable objects are added to the corresponding computation list.

[0011] As a further preferred embodiment of the dynamic computation topology optimization method based on Flink SQL of the present invention, in step 4, the computation process needs to label the data that has been computed, and the computation result data is labeled with the operator ID; the subsequent operators identify which upstream operator the data matches through the data label to determine the subsequent computation context. After the computation is completed, the data is output to a specified number of Sink tables according to the data label.

[0012] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects: This invention presents a dynamic computing topology optimization method based on Flink SQL. It aims to build an innovative optimization method based on the Flink SQL engine, targeting streaming SQL computing scenarios based on the same data source, covering typical computing modes such as data extraction and window aggregation. It enables the dynamic combination, adjustment, and removal of multiple SQL computing operators, including Project, Filter, Aggregate, and Window, within a unified computing topology. Through an intelligent resource scheduling mechanism, it achieves the sharing and reuse of computing resources, effectively supporting the dynamic evolution of real-time SQL logic and significantly improving computing efficiency and resource utilization. Attached Figure Description

[0013] Figure 1 This is a flowchart of the dynamic compression of topology calculation in this invention; Figure 2 This is a schematic diagram of the extraction execution topology of the present invention; Figure 3 This is the compressed computing topology that is currently in operation according to this invention; Figure 4 This is the computational topology after dynamic calculation and merging in this invention. Detailed Implementation

[0014] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings: The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. The present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments. The purpose and effects of the present invention will become clearer. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0015] This invention aims to build an innovative optimization method based on the Flink SQL engine. For streaming SQL computation scenarios based on the same data source, covering typical computational patterns such as data extraction and window aggregation, it enables the dynamic combination, adjustment, and removal of various SQL computation operators, including Project, Filter, Aggregate, and Window, within a unified computational topology. Through an intelligent resource scheduling mechanism, it achieves resource sharing and reuse, effectively supporting the dynamic evolution of real-time SQL logic and significantly improving computational efficiency and resource utilization.

[0016] The relevant concepts of this invention are as follows: Source: The data source defined in the Flink SQL computing framework.

[0017] Sink: Data output defined in the Flink SQL computing framework.

[0018] Job: In the Flink SQL computing engine, a distributed data processing task unit described using the SQL language.

[0019] Operator Chain; In the Flink SQL execution engine, it is an optimized computation that merges multiple logically independent computation operators into a single physical execution unit, Job, under specific conditions.

[0020] TaskManager: A worker node in the Flink SQL execution engine, responsible for the execution of specific tasks and resource management.

[0021] The results of the computational topology dynamic compression optimization method are as follows: Figure 1 As shown: Perform SQL rule matching, such as matching task template IDs using hints, or matching SQL based on the same source (i.e., extraction SQL with the same source and aggregation SQL with the same aggregation conditions). If the current execution plan is an execution plan of a system-defined template type, such as an extraction task or an aggregation task, then the conditions for merging computational operators are initially met. If none of the above conditions are met, the optimization process ends.

[0022] If the above conditions are met, SQL operators are extracted, such as Project, Filter, Aggregate, and Window, to determine if a computation topology with the same source is already running. If not, a new computation topology of that type is started. If a computation topology with the same source already exists, subsequent computational dynamic update operations are performed.

[0023] If dynamic updates to the calculation are required, firstly, code generation operations are performed on the relevant operators of the SQL to generate code descriptions related to these operators, and these code snippets are encapsulated into a Calculate object. At the same time, the object is serialized and sent to the specified TaskManager worker node.

[0024] The worker node receives the Calculate object that needs updating and performs a serialization operation on it. It then parses the object and extracts the operator descriptions, such as Project, Filter, Aggregate, and Window. Finally, it uses the code snippets related to the operators to regenerate the code for the relevant computed objects. The predicate code template is PredicateCalc.

[0025] The code template for projection classes is ProjectCalc.

[0026] The code template for aggregate classes is AggregateCalc.

[0027] The generated code is compiled, and the computable objects are added to the corresponding computation list.

[0028] The computation process requires labeling the completed data, and the computation results are labeled with operator IDs. Subsequent operators use these labels to identify which upstream operator the data matches, thus determining the subsequent computation context.

[0029] After the final calculation is completed, the data is output to a specified number of Sink tables based on the data labels.

[0030] The optimized computation topology description is as follows: Figure 2 As shown: Figure 3The compressed computing topology that is currently in operation is as follows: Figure 3 As shown, Figure 4 Dynamically calculate the merged computation topology as follows Figure 4 As shown.

[0031] As can be seen from the above comparison, this optimization method can dynamically merge submitted SQL tasks of the same type while maintaining the dynamic operation of the Flink computing topology, thus achieving more flexible and efficient computing topology management and computing resource utilization.

[0032] The ability to dynamically update the Flink SQL computation topology in specific scenarios: (1) Dynamic topology classification mechanism: For the same source data extraction scenario and the same source window aggregation scenario, establish the mapping rules between SQL jobs and computing topology, and intelligently classify SQL jobs submitted at different times into the same type of computing topology to form a dynamic topology structure with shared computing resources.

[0033] (2) Dynamic reconstruction of running topology: When it is necessary to terminate a specific SQL job, the calculation expression corresponding to the SQL can be accurately deleted from the running topology through the topology node removal technology, without stopping the operation of the entire topology.

[0034] (3) Dynamic compression and update capability: Break through the static topology limitation of native Flink SQL, realize dynamic compression of computing topology, i.e. resource reclamation and logical update, i.e. computational expression replacement, and support flexible scheduling of computing resources and real-time adjustment of business logic.

[0035] 2) Implement dynamic updates of computational logic (1) Computational logic version management: Build a version control system for SQL expressions, record the computational logic and its dependencies at different stages, and support runtime version switching.

[0036] (2) Runtime logic replacement mechanism: While maintaining the topology running state, the computational logic of the topology nodes is dynamically updated through the logic replacement engine without interrupting the data flow processing.

[0037] (3) State consistency guarantee: Incremental state synchronization technology is adopted to ensure the integrity and consistency of data state during the logic update process, and to avoid data loss or duplicate processing due to logic changes.

[0038] Dynamic compression of the computation topology was achieved in specific scenarios. Apache Flink SQL uses Operator Chain technology to statically compile multiple SQL statements in the same submission, allowing these SQL statements to share computing resources through the same topology. However, this invention can dynamically compress the topology of SQL statements submitted at different times, avoiding the overhead of stopping the topology to share resources.

[0039] Dynamic updates of computational logic within the computational topology were implemented in specific scenarios. In Apache Flink SQL, running computation logic cannot be dynamically updated. This invention enables dynamic updates of computation logic in a running topology under specific scenarios, providing a method for dynamic data debugging and dynamic business updates.

[0040] Dynamically compress the computation topology to reduce resource overhead: Traditional Flink SQL uses Operator Chain to statically compile multiple SQL queries, but this requires binding the topology structure at submission time and cannot be dynamically adjusted. This invention utilizes dynamic topology reconstruction technology to dynamically integrate SQL jobs submitted at different times, achieving: Elastic resource reclamation: When terminating an SQL job, only its corresponding node is removed, avoiding resource waste caused by overall topology downtime; Improved computational efficiency: By sharing computing resources, the overhead of repeated topology initialization can be reduced, and the system throughput can be increased to supplement data support.

[0041] Dynamically update the calculation logic to support real-time adjustments to business operations: The computation logic of existing Flink SQL cannot be dynamically modified after deployment, requiring job resubmission. This invention solves this problem through the following advantages: Downtime update: Update the computing logic without interrupting the data flow to meet the needs of real-time business debugging; Flexible business response: Supports dynamic adjustment of business rules such as window size and aggregation logic to adapt to rapidly changing business scenarios; Reduce maintenance costs: Avoid job restarts and data reprocessing caused by logical changes, and improve system operation and maintenance efficiency.

[0042] It will be understood by those skilled in the art that the above descriptions are merely preferred embodiments of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the invention should be included within the scope of protection of the invention. All technical features in this embodiment can be freely combined according to actual needs.

[0043] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A dynamic computational topology optimization method based on Flink SQL, characterized in that: By matching the native Flink SQL execution plan, a custom computation topology orchestration is reimplemented to achieve dynamic compression of different SQL computation topologies; the specific steps include the following: Step 1: Perform SQL rule matching to determine whether the current SQL meets the topology merging conditions; Step 2: Perform SQL parsing, extract SQL operators, reconstruct the computation object, and serialize it; Step 3: The computation object is sent to the worker node TaskManager. The worker node TaskManager receives the computation object, compiles the object, compiles the computation logic using dynamic code compilation technology, and adds the current computation operator to the computation list. Step 4: After the data calculation is completed, it is sent to the computing unit for projection and final output.

2. The dynamic computation topology optimization method based on Flink SQL according to claim 1, characterized in that: In the steps, task template ID matching is performed by hint matching, and extraction SQL with the same source and aggregation SQL with the same aggregation conditions are matched by same source SQL. If the current execution plan is an execution plan of a template type defined by the system, and is either an extraction task or an aggregation task, then the calculation operator merging condition is met.

3. The dynamic computation topology optimization method based on Flink SQL according to claim 1, characterized in that: In step 2, the SQL operator includes Project, Filter, Aggregate, and Window. It checks whether a computation topology with the same source is already running. If not, a new computation topology is started. If a computation topology with the same source already exists, subsequent dynamic computation update operations are performed. If dynamic computation update is required, the relevant SQL operators are first processed to generate code descriptions for these operators. These code snippets are then encapsulated into a Calculate object, and the object is serialized and sent to the specified worker node TaskManager.

4. The dynamic computation topology optimization method based on Flink SQL according to claim 3, characterized in that: In step 3, the worker node TaskManager receives the Calculate object that needs to be updated, performs a serialization operation on the Calculate object, parses the Calculate object and extracts the operator descriptions, including Project, Filter, Aggregate, and Window; and regenerates the code for the relevant calculation object using the code snippets of the relevant operators. The predicate code template is PredicateCalc; The code template for projection classes is ProjectCalc; The code template for aggregation classes is AggregateCalc; The generated code is compiled, and the computable objects are added to the corresponding computation list.

5. The dynamic computation topology optimization method based on Flink SQL according to claim 1, characterized in that: In step 4, the data that has been calculated is labeled, and the calculation result data is labeled with the operator ID; subsequent operators identify whether the data matches the upstream operator through the data label, and thus determine the subsequent calculation context. After the calculation is completed, the data is output to a specified number of Sink tables according to the data label.

Images